counterweight/superhog-infra-repo
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

change section numbering

Browse source
This commit is contained in:
Pablo Martin 2024-11-26 11:13:18 +01:00
parent 8b1c8b5c9c
commit 94191161b6
1 changed files with 36 additions and 64 deletions
Download patch file Download diff file Expand all files Collapse all files

100
human-script.md
View file

@ -2,14 +2,14 @@
Follow this to deploy the entire data infra. Follow this to deploy the entire data infra.
## 0. Pre-requisites and conventions ## 000. Pre-requisites and conventions
- You need an Azure subscription and a user with administrator rights in it. - You need an Azure subscription and a user with administrator rights in it.
- Whenever you see `<your-env>`, you should replace that with `dev`,`uat`, `prd` or whatever fits your environment. - Whenever you see `<your-env>`, you should replace that with `dev`,`uat`, `prd` or whatever fits your environment.
- We traditionally deploy resources on the `UK South` region. Unless stated otherwise, you should deploy resources there. - We traditionally deploy resources on the `UK South` region. Unless stated otherwise, you should deploy resources there.
- You have an SSH key pair ready to use for access to the different machines. You can always add more pairs later. - You have an SSH key pair ready to use for access to the different machines. You can always add more pairs later.
## 1. Resource group and SSH Keypair ## 010. Resource group and SSH Keypair
### 1.1 Create Resource Group ### 1.1 Create Resource Group
@ -30,9 +30,9 @@ Follow this to deploy the entire data infra.
- Pay attention when storing the private key. You probably want to store it in a safe password manager, like Keeper. - Pay attention when storing the private key. You probably want to store it in a safe password manager, like Keeper.
- Optionally, you can also be extra paranoid, generate the SSH key locally and only upload the public key to Azure. Up to you. - Optionally, you can also be extra paranoid, generate the SSH key locally and only upload the public key to Azure. Up to you.
## 2. Networking ## 020. Networking
### 2.1 VNET ### 020.1 VNET
- Create a virtual network. This virtual network is where all our infra will live. For the rest of this guide, assume this is the network where you must connect services. - Create a virtual network. This virtual network is where all our infra will live. For the rest of this guide, assume this is the network where you must connect services.
- Name it: `superhog-data-vnet-<your-env>` - Name it: `superhog-data-vnet-<your-env>`
@ -60,7 +60,7 @@ Follow this to deploy the entire data infra.
- `environment: <your-env>` - `environment: <your-env>`
- `project: network` - `project: network`
### 2.2 Network security groups ### 020.2 Network security groups
- You will create three network security groups (NSG) - You will create three network security groups (NSG)
- Jumphost NSG - Jumphost NSG
@ -172,7 +172,7 @@ Follow this to deploy the entire data infra.
- Visit the virtual network page and look for the subnets list - Visit the virtual network page and look for the subnets list
- For each subnet, select its NSG and attach it - For each subnet, select its NSG and attach it
### 2.3 Private DNS Zone ### 020.3 Private DNS Zone
- We will set up a private DNS Zone to avoid using hardcoded IPs to refer to services within the virtual network. This makes integrations more resilient because a service can change its IP and still be reached by other services (as long as other network configs like firewalls are still fine). - We will set up a private DNS Zone to avoid using hardcoded IPs to refer to services within the virtual network. This makes integrations more resilient because a service can change its IP and still be reached by other services (as long as other network configs like firewalls are still fine).
- Create the Private DNS Zone - Create the Private DNS Zone
@ -186,7 +186,7 @@ Follow this to deploy the entire data infra.
- Associate it to the virtual network. - Associate it to the virtual network.
- Enable autoregistration - Enable autoregistration
### 2.4 Public IP ### 020.4 Public IP
- We will need a public IP for the jumphost. - We will need a public IP for the jumphost.
- Create the public IP - Create the public IP
@ -197,9 +197,9 @@ Follow this to deploy the entire data infra.
- `environment: <your-env>` - `environment: <your-env>`
- `project: network` - `project: network`
## 3. Jumphost ## 030. Jumphost
### 3.1 Deploy Jumphost VM ### 030.1 Deploy Jumphost VM
- The first VM we must deploy is a jumphost, since that will be our door to all other services inside the virtual network. - The first VM we must deploy is a jumphost, since that will be our door to all other services inside the virtual network.
- Create the VM - Create the VM
@ -228,7 +228,7 @@ Follow this to deploy the entire data infra.
- `environment: <your-env>` - `environment: <your-env>`
- `project: network` - `project: network`
### 3.2 Configure a VPN Server ### 030.2 Configure a VPN Server
- The jumphost we just created is not accessible via SSH from WAN due to the NSG set in the jumphost subnet. - The jumphost we just created is not accessible via SSH from WAN due to the NSG set in the jumphost subnet.
- To make it so, you should temporarily create a new rule like this in the NSG `superhog-data-nsg-jumphost-<your-env>`. - To make it so, you should temporarily create a new rule like this in the NSG `superhog-data-nsg-jumphost-<your-env>`.
@ -322,7 +322,7 @@ Follow this to deploy the entire data infra.
- Look for the jumphost VM Network Interface. - Look for the jumphost VM Network Interface.
- In the `IP configurations` session, activate the flag `Enable IP forwarding`. - In the `IP configurations` session, activate the flag `Enable IP forwarding`.
### 3.3 Configure a DNS Server ### 030.3 Configure a DNS Server
- The jumphost is now ready. When the VPN is active on our local device, we can access the services within the virtual network. - The jumphost is now ready. When the VPN is active on our local device, we can access the services within the virtual network.
- There is one issue, though: we would like to access services through names, not IPs. - There is one issue, though: we would like to access services through names, not IPs.
@ -379,14 +379,14 @@ Follow this to deploy the entire data infra.
- In your client Wireguard configuration, uncomment the DNS server line we left before - In your client Wireguard configuration, uncomment the DNS server line we left before
- Check that the service is running fine by running `dig google.com`. You should see in the output that your laptop has relied on our new DNS to do the name resolution. - Check that the service is running fine by running `dig google.com`. You should see in the output that your laptop has relied on our new DNS to do the name resolution.
### 3.4 Harden the Jumphost VM ### 030.4 Harden the Jumphost VM
- In the Jumphost, run the following command to disable password based SSH authentication fully. This way, access can only be granted with SSH key pairs, which is way more secure: `sudo sed -i -e 's/#PasswordAuthentication yes/PasswordAuthentication no/g' /etc/ssh/sshd_config; sudo systemctl restart ssh`. - In the Jumphost, run the following command to disable password based SSH authentication fully. This way, access can only be granted with SSH key pairs, which is way more secure: `sudo sed -i -e 's/#PasswordAuthentication yes/PasswordAuthentication no/g' /etc/ssh/sshd_config; sudo systemctl restart ssh`.
- Remove the AllowSSHInboundTemporarily rule that you added to the NSG `superhog-data-nsg-jumphost-<your-env>`. We don't need that anymore since we can SSH through the VPN tunnel. - Remove the AllowSSHInboundTemporarily rule that you added to the NSG `superhog-data-nsg-jumphost-<your-env>`. We don't need that anymore since we can SSH through the VPN tunnel.
## 4. DWH ## 040. DWH
### 4.1 Deploy PostgreSQL Server ### 040.1 Deploy PostgreSQL Server
- Next, we will deploy a Postgres server to act as the DWH. - Next, we will deploy a Postgres server to act as the DWH.
- Create a new Azure Database for PostgreSQL flexible servers. - Create a new Azure Database for PostgreSQL flexible servers.
@ -410,7 +410,7 @@ Follow this to deploy the entire data infra.
- Validate the deployment by trying to log into the database with the `dwh_admin_<your-env>` user from your favourite SQL client (you can use DBeaver, for example). Be aware that your VPN connection should be active so that the DWH is reachable from your device. - Validate the deployment by trying to log into the database with the `dwh_admin_<your-env>` user from your favourite SQL client (you can use DBeaver, for example). Be aware that your VPN connection should be active so that the DWH is reachable from your device.
### 4.2 Create database ### 040.2 Create database
- Run the following commands to create a new database - Run the following commands to create a new database
@ -420,7 +420,7 @@ Follow this to deploy the entire data infra.
- From now on, use this database for everything - From now on, use this database for everything
### 4.3 Create schemas, roles and users ### 040.3 Create schemas, roles and users
- Run the following script to create: - Run the following script to create:
- A `dwh_builder` role, which: - A `dwh_builder` role, which:
@ -515,11 +515,11 @@ Follow this to deploy the entire data infra.
ALTER DEFAULT PRIVILEGES IN SCHEMA sync_<some-new-source> GRANT SELECT ON TABLES TO modeler; ALTER DEFAULT PRIVILEGES IN SCHEMA sync_<some-new-source> GRANT SELECT ON TABLES TO modeler;
``` ```
## 5. Web Gateway ## 050. Web Gateway
We will deploy a dedicated VM to act as a web server for internal services. We will deploy a dedicated VM to act as a web server for internal services.
### 5.1 Deploy Web Gateway VM ### 050.1 Deploy Web Gateway VM
- Create a new VM following these steps. - Create a new VM following these steps.
- Basic settings - Basic settings
@ -548,7 +548,7 @@ We will deploy a dedicated VM to act as a web server for internal services.
- `project: network` - `project: network`
- Once the VM is running, you should be able to ssh into the machine when your VPN is active. - Once the VM is running, you should be able to ssh into the machine when your VPN is active.
### 9.2 Deploying Caddy ### 050.2 Deploying Caddy
- We need to install caddy in the VM. You can do so with the following commands: - We need to install caddy in the VM. You can do so with the following commands:
@ -563,7 +563,7 @@ We will deploy a dedicated VM to act as a web server for internal services.
- After the previous commands, you can verify that caddy is running properly as a systemd service with: `systemctl status caddy` - After the previous commands, you can verify that caddy is running properly as a systemd service with: `systemctl status caddy`
- You can also verify that Caddy is reachable (should be) by running the following command from your laptop while connected to the VPN: `curl web-gateway-<your-env>.<your-env>.data.superhog.com`. If you see a wall of HTML that looks like Caddy's demo page, it means Caddy is working as expected. - You can also verify that Caddy is reachable (should be) by running the following command from your laptop while connected to the VPN: `curl web-gateway-<your-env>.<your-env>.data.superhog.com`. If you see a wall of HTML that looks like Caddy's demo page, it means Caddy is working as expected.
### 9.3 Pointing Caddy to internal services ### 050.3 Pointing Caddy to internal services
- Caddy will need to be configured to act as the web server or reverse proxy of the different services within the services subnet. The details of these configurations are defined in sections below. - Caddy will need to be configured to act as the web server or reverse proxy of the different services within the services subnet. The details of these configurations are defined in sections below.
- As a general note, the pattern will generally be: - As a general note, the pattern will generally be:
@ -571,9 +571,9 @@ We will deploy a dedicated VM to act as a web server for internal services.
- You will need to reload caddy with `sudo systemctl reload caddy.service`. - You will need to reload caddy with `sudo systemctl reload caddy.service`.
- If the web server needs to reach a specific port in some other VM, you will need to sort networking security out. If the VM you need to reach from the web server is within the internal services subnet, you'll have to add the necessary Inbound rules in the NSG `superhog-data-nsg-services-<your-env>`. - If the web server needs to reach a specific port in some other VM, you will need to sort networking security out. If the VM you need to reach from the web server is within the internal services subnet, you'll have to add the necessary Inbound rules in the NSG `superhog-data-nsg-services-<your-env>`.
## 5. Airbyte ## 060. Airbyte
### 5.1 Deploying Airbyte VM ### 060.1 Deploying Airbyte VM
- Airbyte lives on its own VM. To do so, create a new VM following these steps. - Airbyte lives on its own VM. To do so, create a new VM following these steps.
- Basic settings - Basic settings
@ -602,7 +602,7 @@ We will deploy a dedicated VM to act as a web server for internal services.
- `project: airbyte` - `project: airbyte`
- Once the VM is running, you should be able to ssh into the machine when your VPN is active. - Once the VM is running, you should be able to ssh into the machine when your VPN is active.
### 5.2 Deploying Airbyte ### 060.2 Deploying Airbyte
- SSH into the VM. - SSH into the VM.
- Run the following script to install docker and deploy Airbyte - Run the following script to install docker and deploy Airbyte
@ -646,7 +646,7 @@ We will deploy a dedicated VM to act as a web server for internal services.
- To check that Airbyte is running fine, run this command from a terminal within the Airbyte VM: `curl localhost:8000`. You should see some HTML for Airbyte's access denied page. - To check that Airbyte is running fine, run this command from a terminal within the Airbyte VM: `curl localhost:8000`. You should see some HTML for Airbyte's access denied page.
### 5.3 Making Airbyte Web UI reachable ### 060.3 Making Airbyte Web UI reachable
- To provide access to the Airbyte UI, we will have to integrate it with the web gateway and our networking configurations. - To provide access to the Airbyte UI, we will have to integrate it with the web gateway and our networking configurations.
- First, we need to allow the web gateway to reach Airbyte locally-served webserver. - First, we need to allow the web gateway to reach Airbyte locally-served webserver.
@ -697,9 +697,9 @@ We will deploy a dedicated VM to act as a web server for internal services.
- If everything is working properly, you should now be able to reach airbyte at the printed URL. - If everything is working properly, you should now be able to reach airbyte at the printed URL.
- If something doesn't work, I would advise troubleshooting through the chain of machines (your device to the VPN box, then to the web gateway, then to the airbyte machine) to find where is the connection breaking down. - If something doesn't work, I would advise troubleshooting through the chain of machines (your device to the VPN box, then to the web gateway, then to the airbyte machine) to find where is the connection breaking down.
## 6. Power BI ## 070. Power BI
### 6.1 Deploying Power BI VM ### 070.1 Deploying Power BI VM
- We need to deploy a Windows VM. - We need to deploy a Windows VM.
- Create the VM - Create the VM
@ -729,7 +729,7 @@ We will deploy a dedicated VM to act as a web server for internal services.
- `project: pbi` - `project: pbi`
- Try to connect with RDP at `pbi-gateway-<your-env>.<your-env>.data.superhog.com`. - Try to connect with RDP at `pbi-gateway-<your-env>.<your-env>.data.superhog.com`.
### 6.2 Installing Power BI Data Gateway ### 070.2 Installing Power BI Data Gateway
- Login the VM. - Login the VM.
- Follow the instructions here to download the installer in the VM and set it up: <https://learn.microsoft.com/en-us/data-integration/gateway/service-gateway-install> - Follow the instructions here to download the installer in the VM and set it up: <https://learn.microsoft.com/en-us/data-integration/gateway/service-gateway-install>
@ -745,68 +745,40 @@ We will deploy a dedicated VM to act as a web server for internal services.
- Turn the `require_secure_transport` parameter to `Off`. - Turn the `require_secure_transport` parameter to `Off`.
- Once you are done, you should be able to visit the PBI Service (the online UI), visit the gateways page in settings and see the gateway listed in the `On-premises data gateways` section. - Once you are done, you should be able to visit the PBI Service (the online UI), visit the gateways page in settings and see the gateway listed in the `On-premises data gateways` section.
## 7. dbt ## 080. dbt
- Our dbt project (<https://guardhog.visualstudio.com/Data/_git/data-dwh-dbt-project>) can be deployed on any linux VM within the virtual network. The instructions on how to deploy and schedule it are in the project repository. - Our dbt project (<https://guardhog.visualstudio.com/Data/_git/data-dwh-dbt-project>) can be deployed on any linux VM within the virtual network. The instructions on how to deploy and schedule it are in the project repository.
- You can opt to deploy it in the same machine where airbyte is stored, since that machine is probably fairly underutilized. - You can opt to deploy it in the same machine where airbyte is stored, since that machine is probably fairly underutilized.
## 8. Monitoring ## 090. Monitoring
### 8.1 Infra monitoring ### 090.1 Infra monitoring
WIP: we are planning on using Azure Dashboards with metrics. WIP: we are planning on using Azure Dashboards with metrics.
### 8.2 Service status ### 090.2 Service status
WIP: we need support to learn how to use statuspage.io WIP: we need support to learn how to use statuspage.io
### 9.3 Configuring Caddy
- Now that caddy is running, you can configure it to serve whatever you need. ## 100. Backups
- This instance is designed to be the external entrypoint to serve any internal webpages to users of web services within the data virtual network. It's possible that, by the time you are reading this, there are more services that we planned originally.
- As an example, we will now show how to reverse proxy the Airbyte UI. For other services, you can follow a similar pattern.
- Edit the caddy config file with `sudo nano /etc/caddy/Caddyfile`
- To add a reverse proxy for Airbyte, add this entry:
```bash
http://airbyte.prd.data.superhog.com {
reverse_proxy http://airbyte-<your-env>.<your-env>.data.superhog.com {
#reverse_proxy http://10.69.0.68:80 {
header_up Cookie {>Cookie}
header_up Host airbyte-prd.prd.data.superhog.com
header_up X-Real-IP {remote}
header_up X-Forwarded-For {remote}
header_up X-Forwarded-Proto {scheme}
}
}
```
- Note that, if you need to do more changes in configuration, you can have Caddy pick up the changes by running `sudo systemctl reload caddy`. This will reload the configuration without incurring any downtime, as `stop` and `start` would.
### 9.4 Additional networking actions
- Allow internal service VMs to reach each other at port 80, it's in the NSG for the services subnet
## 10. Backups
- If you are working on a dev or staging environment, you might want to skip this section. - If you are working on a dev or staging environment, you might want to skip this section.
### 10.1 DWH ### 100.1 DWH
- Backups are managed with Azure. In the Azure Portal page for the PostgreSQL service, visit section `Backup and restore`. Production servers should have 14 days as a retention period. - Backups are managed with Azure. In the Azure Portal page for the PostgreSQL service, visit section `Backup and restore`. Production servers should have 14 days as a retention period.
### 10.2 Jumphost ### 100.2 Jumphost
- Jumphosts barely hold any data at all. Although it's quite tempting to forget about this and simply raise another VM if something goes wrong, it would be annoying to have to regenerate the keys of both the VPN server and other clients. - Jumphosts barely hold any data at all. Although it's quite tempting to forget about this and simply raise another VM if something goes wrong, it would be annoying to have to regenerate the keys of both the VPN server and other clients.
- To solve this, make a habit of making regular copies of the Wireguard config file in another machine. Theoretically, only making a copy everytime it gets modified should be enough. - To solve this, make a habit of making regular copies of the Wireguard config file in another machine. Theoretically, only making a copy everytime it gets modified should be enough.
### 10.3 Airbyte ### 100.3 Airbyte
- Our strategy for backing up Airbyte is to backup the entire VM. - Our strategy for backing up Airbyte is to backup the entire VM.
- WIP - WIP
### 10.4 PBI Gateway ### 100.4 PBI Gateway
- The PBI Gateway is pretty much stateless. Given this, if there are any issues or disasters on the current VM, simply create another one and set up the gateway again. - The PBI Gateway is pretty much stateless. Given this, if there are any issues or disasters on the current VM, simply create another one and set up the gateway again.