Merged PR 2669: Scheduled tests

# Description

This PR provides a suggested script to run `dbt test` in production, leave logs and send slack messages upon success/failure. It also updates the readme details on how to deploy so that deployers know how to set it.

# Checklist

- __NA__ ~~[ ] The edited models and dependants run properly with production data.~~
- __NA__ ~~[ ] The edited models are sufficiently documented.~~
- __NA__ ~~[ ] The edited models contain PK tests, and I've ran and passed them.~~
- __NA__ ~~[ ] I have checked for DRY opportunities with other models and docs.~~
- __NA__ ~~[ ] I've picked the right materialization for the affected models.~~

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #16959

README.md

@@ -113,13 +113,14 @@ To deploy:
 - Activate the virtual environment and run `pip install -r requirements.txt`
 - Also run `dbt deps` to install the dbt packages required by the project.
 - Create an entry for this project `profiles.yml` file at `~/.dbt/profiles.yml`. You have a suggested template at `profiles.yml.example`. Make sure that the `profiles.yml` host and port settings are consistent with whatever networking approach you've taken.
-- There's a script in the root of this project called `run_dbt.sh`. Place it in `~/run_dbt.sh`. Adjust the paths of the script if you want/need to.
-- The script is designed to send both success and failure messages to slack channels upon completion. To properly set this up, you will need to place a file called `slack_webhook_urls.txt` on the same path you drop `run_dbt.sh`. The file should have two lines: `SLACK_ALERT_WEBHOOK_URL=<url-of-webhook-for-failures>` and `SLACK_RECEIPT_WEBHOOK_URL=<url-of-webhook-for-successful-runs>`. Setting up the slack channels  and webhooks is outside of the scope of this readme.
-- Create a cron entry with `crontab -e` that runs the script. For example: `0 2 * * * /bin/bash /home/azureuser/run_dbt.sh` to run the dbt models every day at 2AM.
+- There are two scripts in the root of this project called `run_dbt.sh` and `run_tests.sh`. Place them in the running user's home folder. Adjust the paths of the script if you want/need to.
+- The scripts are designed to send both success and failure messages to slack channels upon completion. To properly set this up, you will need to place a file called `slack_webhook_urls.txt` on the same path you put the script files. The slack webhooks file should have two lines: `SLACK_ALERT_WEBHOOK_URL=<url-of-webhook-for-failures>` and `SLACK_RECEIPT_WEBHOOK_URL=<url-of-webhook-for-successful-runs>`. Setting up the slack channels and webhooks is outside of the scope of this readme.
+- Create a cron entry with `crontab -e` that runs the scripts. For example: `0 2 * * * /bin/bash /home/azureuser/run_dbt.sh` to run the dbt models every day at 2AM, and `15 2 * * * /bin/bash /home/azureuser/run_tests.sh` to run the tests fifteen minutes later.
 
 To monitor:
 
-- The script writes output to a `dbt_run.log` file. You can check the contents to see what happened in the past runs. The exact location of the log file depends on how you set up the `run_dbt.sh` script. If you are unsure of where your logs are being written, check the script to find out.
+- The model building script writes output to a `dbt_run.log` file. You can check the contents to see what happened in the past runs. The exact location of the log file depends on how you set up the `run_dbt.sh` script. If you are unsure of where your logs are being written, check the script to find out.
+- Same applies to the test script, except it will write into a separate `dbt_test.log`.
 
 To maintain:
 

run_tests.sh

@@ -0,0 +1,46 @@
+#!/bin/bash
+
+exec >> /home/azureuser/dbt_tests.log 2>&1
+
+# Define the Slack webhook URL
+script_dir=$(dirname "$0")
+webhooks_file="slack_webhook_urls.txt"
+env_file="$script_dir/$webhooks_file"
+if [ -f "$env_file" ]; then
+  export $(grep -v '^#' "$env_file" | xargs)
+else
+  echo "Error: $webhooks_file file not found in the script directory."
+  exit 1
+fi
+
+# Messages to be sent to Slack
+slack_failure_message=":rotating_light::rotating_light::rotating_light: One or more failures in dbt tests in production. :rotating_light::rotating_light::rotating_light:"
+slack_success_message=":white_check_mark::white_check_mark::white_check_mark: dbt tests executed successfully in production. :white_check_mark::white_check_mark::white_check_mark:"
+
+# Initialize the failure flag
+has_any_step_failed=0
+
+cd /home/azureuser/data-dwh-dbt-project
+
+# Update from git
+echo "Updating dbt project from git."
+git checkout master
+git pull
+
+# Activate venv
+source venv/bin/activate
+
+# Run tests
+echo "Triggering dbt test"
+dbt test
+if [ $? -ne 0 ]; then
+  has_any_step_failed=1
+fi
+
+# Check if any step failed and send a Slack message
+if [ $has_any_step_failed -eq 1 ]; then
+  curl -X POST -H 'Content-type: application/json' --data "{\"text\":\"$slack_failure_message\"}" $SLACK_ALERT_WEBHOOK_URL
+fi
+if [ $has_any_step_failed -eq 0 ]; then
+  curl -X POST -H 'Content-type: application/json' --data "{\"text\":\"$slack_success_message\"}" $SLACK_RECEIPT_WEBHOOK_URL
+fi