counterweight/data-xexe
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

more readme instructions

Browse source
This commit is contained in:
Pablo Martin 2024-06-13 19:24:57 +02:00
parent f21c0c187b
commit ae3ad75073
1 changed files with 4 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

6
README.md
View file

@ -145,10 +145,12 @@ This tool was made specifically to feed our DWH. These are the steps to perform
- `How to Use the tool > Set up credentials`
- `How to Use the tool > DWH pre-requisites`
- Schedule
- Finally, schedule the execution in the Linux VM to fit your needs.
- Up next, schedule the execution in the Linux VM to fit your needs.
- Specifics are up to you and your circunstances.
- A general pattern would be to create a little bash script that calls the tool with the right parameters on it. You can find an example that I like in the root of this repo named `run_xexe.sh`, but that's opinionated and adjusted to my needs at the time of writing this. Adapt it to your environment or start from scratch if necessary.
- A general pattern would be to create a little bash script that calls the tool with the right parameters on it. You can find an example that I like in the root of this repo named `run_xexe.sh`, but that's opinionated and adjusted to my needs at the time of writing this. Adapt it to your environment or start from scratch if necessary. The script is designed to be placed in`~/run_xexe.sh`.
- Remember to use the `--ignore-warnings` flag if necessary to allow large, automated runs without manually interaction.
- The script is designed to send both success and failure messages to slack channels upon completion. To properly set this up, you will need to place a file called `slack_webhook_urls.txt` on the same path you drop `run_xexe.sh`. The file should have two lines: `SLACK_ALERT_WEBHOOK_URL=<url-of-webhook-for-failures>` and `SLACK_RECEIPT_WEBHOOK_URL=<url-of-webhook-for-successful-runs>`. Setting up the slack channels and webhooks is outside of the scope of this readme.
- Create a cron entry with `crontab -e` that runs the script. For example: `0 2 * * * /bin/bash /home/azureuser/run_xexe.sh` to run xexe models every day at 2AM.
- Backfilling
- If you are loading rates for the first time, you might need to backfill long periods of time manually at first.
- The tool is flexible enough. You can probably figure out the right commands by taking a look at `How to Use the tool > General Usage`