64 lines
No EOL
1.8 KiB
Markdown
64 lines
No EOL
1.8 KiB
Markdown
# Lolafect
|
|
|
|
Lolafect is a collection of Python bits that help us build our Prefect flows.
|
|
|
|
## Quickstart
|
|
|
|
You can find below examples of how to leverage `lolafect` in your flows.
|
|
|
|
**Let the `LolaConfig` object do the boilerplate env stuff for you**
|
|
|
|
```python
|
|
from lolafect.lolaconfig import build_lolaconfig
|
|
|
|
lolaconfig = build_lolaconfig(
|
|
flow_name="some-flow",
|
|
env_s3_bucket="bucket",
|
|
kubernetes_labels=["some_label"],
|
|
kubernetes_image="the-image:latest",
|
|
)
|
|
|
|
# Now you can access all the env stuff from here
|
|
lolaconfig.FLOW_NAME
|
|
lolaconfig.FLOW_NAME_UDCS
|
|
lolaconfig.STORAGE
|
|
lolaconfig.KUBERNETES_IMAGE
|
|
lolaconfig.KUBERNETES_LABELS
|
|
lolaconfig.SLACK_WEBHOOKS
|
|
lolaconfig.DW_CREDENTIALS
|
|
lolaconfig.TRINO_CREDENTIALS
|
|
lolaconfig.SSH_TUNNEL_CREDENTIALS
|
|
lolaconfig.PREFECT_HOST
|
|
# etc
|
|
```
|
|
|
|
**Send a warning message to slack if your tasks fails**
|
|
|
|
```python
|
|
from prefect.triggers import any_failed
|
|
from lolafect.slack import SendSlackMessageTask
|
|
|
|
send_warning_message_on_any_failure = SendSlackMessageTask(trigger=any_failed) # You can generate other tasks with
|
|
#different triggers. For example, you can send a message when all tasks fail, or all tasks succeed
|
|
|
|
with Flow(...) as flow:
|
|
crucial_task_result = some_crucial_task()
|
|
|
|
send_warning_message_on_any_failure(
|
|
webhook_url="the-channel-webhook", # You can probably try to fetch this from lolaconfig.SLACK_WEBHOOKS
|
|
text_to_send="Watchout, the flow failed!",
|
|
upstream_tasks=[crucial_task_result]
|
|
)
|
|
```
|
|
|
|
## How to test
|
|
|
|
IDE-agnostic:
|
|
|
|
1. Set up a virtual environment which contains both `lolafect` and the dependencies listed in `requirements-dev.txt`.
|
|
2. Run: `pytests tests`
|
|
|
|
In Pycharm:
|
|
|
|
- If you configure `pytest` as the project test runner, Pycharm will most probably autodetect the test
|
|
folder and allow you to run the test suite within the IDE. |