From 8057ab96cecb23493a0fb870b7444fc65e25b130 Mon Sep 17 00:00:00 2001 From: Pablo Martin Date: Mon, 12 Feb 2024 17:44:49 +0100 Subject: [PATCH] Backup stuff --- human-script.md | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/human-script.md b/human-script.md index a240c40..11268c6 100644 --- a/human-script.md +++ b/human-script.md @@ -608,8 +608,37 @@ Follow this to deploy the entire data infra. ## 7. dbt -## 8. Status monitoring +- Our dbt project (https://guardhog.visualstudio.com/Data/_git/data-dwh-dbt-project) can be deployed on any linux VM within the virtual network. The instructions on how to deploy and schedule it are in the project repository. +- You can opt to deploy it in the same machine where airbyte is stored, since that machine is probably fairly underutilized. + +## 8. Monitoring + +### 8.1 Infra monitoring + +WIP: we are planning on using Azure Dashboards with metrics. + +### 8.2 Service status + +WIP: we need support to learn how to use statuspage.io ## 9. Backups - If you are working on a dev or staging environment, you might want to skip this section. + +## 9.1 DWH + +- Backups are managed with Azure. In the Azure Portal page for the PostgreSQL service, visit section `Backup and restore`. Production servers should have 14 days as a retention period. + +## 9.2 Jumphost + +- Jumphosts barely hold any data at all. Although it's quite tempting to forget about this and simply raise another VM if something goes wrong, it would be annoying to have to regenerate the keys of both the VPN server and other clients. +- To solve this, make a habit of making regular copies of the Wireguard config file in another machine. Theoretically, only making a copy everytime it gets modified should be enough. + +## 9.3 Airbyte + +- Our strategy for backing up Airbyte is to backup the entire VM. +- WIP + +## 9.4 PBI Gateway + +- The PBI Gateway is pretty much stateless. Given this, if there are any issues or disasters on the current VM, simply create another one and set up the gateway again.