Lolafect

Lolafect is a collection of Python bits that help us build our Prefect flows.

Quickstart

You can find below examples of how to leverage lolafect in your flows.

Let the LolaConfig object do the boilerplate env stuff for you

from lolafect.lolaconfig import build_lolaconfig

lolaconfig = build_lolaconfig(flow_name="some-flow")

# Now you can access all the env stuff from here
lolaconfig.FLOW_NAME
lolaconfig.FLOW_NAME_UDCS
lolaconfig.STORAGE
lolaconfig.KUBERNETES_IMAGE
lolaconfig.KUBERNETES_LABELS
lolaconfig.SLACK_WEBHOOKS
lolaconfig.DW_CREDENTIALS
lolaconfig.TRINO_CREDENTIALS
lolaconfig.SSH_TUNNEL_CREDENTIALS
lolaconfig.PREFECT_HOST
# etc

# Your flow is different from the typical one?
# You can customize the behaviour of LolaConfig
lolaconfig = build_lolaconfig(
    flow_name="some-flow",
    env_s3_bucket="my-odd-bucket",
    kubernetes_labels=["some-unusual-label"],
    kubernetes_image="the-image:not-the-production-one",
)

Send a warning message to slack if your tasks fails

from prefect.triggers import any_failed
from lolafect.slack import SendSlackMessageTask

send_warning_message_on_any_failure = SendSlackMessageTask(trigger=any_failed) # You can generate other tasks with 
#different triggers. For example, you can send a message when all tasks fail, or all tasks succeed

with Flow(...) as flow:
    crucial_task_result = some_crucial_task()

    send_warning_message_on_any_failure(
        webhook_url="the-channel-webhook", # You can probably try to fetch this from lolaconfig.SLACK_WEBHOOKS
        text_to_send="Watchout, the flow failed!",
        upstream_tasks=[crucial_task_result]
    )

How to test

IDE-agnostic:

1. Set up a virtual environment which contains both lolafect and the dependencies listed in requirements-dev.txt.
2. Run: pytests tests

In Pycharm:

• If you configure pytest as the project test runner, Pycharm will most probably autodetect the test folder and allow you to run the test suite within the IDE.