- Create a resource group. This resource group will hold all the resources. For the rest of this guide, assume this is the resource group where you must create resources.
- Create a virtual network. This virtual network is where all our infra will live. For the rest of this guide, assume this is the network where you must connect services.
- Name it: `superhog-data-vnet-<your-env>`
- You need to think what the network range should be like. For example, you could decide that the entire vnet will be contained within. For reference, we should be fine with a `/24` space (256 addresses) since we will only have a handful network interfaces connecting.
- As an example, we will use `10.69.0.0/24`. This link might be helpful: <https://www.davidc.net/sites/default/subnets/subnets.html?network=10.69.0.0&mask=24&division=11.f10>
- You need to add three subnets:
- Add no network security groups to any of the subnets still. We will create those later.
- Jumphost subnet
- This subnet is where jumphost boxes will live.
- It will be the only subnet where we allow inbound connections from WAN.
- Name it `jumphost-subnet`.
- For our example, we will make it `10.69.0.0/29` (8 addresses).
- Database subnet
- This subnet is where the DWH database will live.
- Inbound traffic will be allowed from both the jumphost subnet as well as the services subnet.
- Name it `database-subnet`
- For our example, we will make it `10.69.0.8/29` (8 addresses).
- Services subnet
- This subnet is where most VMs dedicated to data services live (Airbyte, dbt, PBI Data Gateway, etc).
- Inbound traffic will only be allowed from the jumphost subnet.
- Name it `services-subnet`
- For our example, we will make it `10.69.0.64/26` (64 addresses)
- Add tags:
-`team: data`
-`environment: <your-env>`
-`project: network`
### 2.2 Network security groups
- You will create three network security groups (NSG)
- Jumphost NSG
- Name it: `superhog-data-nsg-jumphost-<your-env>`
- Purpose: only allow connecting to the VPN server. We deny absolutely any other inbound traffic.
- Add tags:
-`team: data`
-`environment: <your-env>`
-`project: network`
- Add the following inbound rules
- VPN Rule
- Name: AllowWireguardInbound
- Source: Any
- Source port ranges: *
- Destination: the addresss range for the `jumphost-subnet`. In this example, `10.69.0.0/29`.
- We will set up a private DNS Zone to avoid using hardcoded IPs to refer to services within the virtual network. This makes integrations more resilient because a service can change its IP and still be reached by other services (as long as other network configs like firewalls are still fine).
- Create the Private DNS Zone
- Name it: `<your-env>.data.superhog.com`
- Add tags:
-`team: data`
-`environment: <your-env>`
-`project: network`
- Add a new virtual network link to the zone
- Name it: `privatelink-<your-env>.data.superhog.com`
- *Note: the IPs chosen for the VPN can absolutely be changed. Just make sure they are consistent across the server and client configurations of the VPN.*
- You should copy the client config that the script will produce and set up the Wireguard config on your local machine.
- Once you've done so, start Wireguard on the client and try to ping the server from the client with the Wireguard VPN IP. If it reaches, the VPN is working fine.
- Now, validate your setup by SSHing from your local device into the jumphost by referencing the VPN IP of the jumphost instead of the public IP.
- Once you verify everything works, you should go to the NSG of the jumphost and remove rule AllowSSHInboundTemporarily. From this point on, the only entrypoint from WAN to the virtual network is the VPN port in the jumphost machine.
- Next, we must allow IP forwarding on Azure.
- Look for the jumphost VM Network Interface.
- In the `IP configurations` session, activate the flag `Enable IP forwarding`.
- The jumphost is now ready. When the VPN is active on our local device, we can access the services within the virtual network.
- There is one issue, though: we would like to access services through names, not IPs.
- Our Private DNS Zone takes care of providing names to services within the virtual network. But these resolution only happens within the virtual network itself, so our external device can't rely on it.
- To solve this, we need to force DNS resolution of our laptops to happen from within the virtual network itself.
- To do so, we will set up a DNS server in the jumphost, and set up our VPN configuration to use it when the VPN connection in our device is active.
- Connect to the jumphost through SSH
- Run the following script as `sudo` from the home folder of `azureuser`
sed -i -e 's/#DNSStubListener=yes/DNSStubListener=no/g' /etc/systemd/resolved.conf
systemctl restart systemd-resolved
echo "Writing config file".
rm /etc/coredns/Corefile
cat > /etc/coredns/Corefile <<EOL
. {
hosts {
log
# If you want to make custom mappings, place them here
# Format is
# xxx.xxx.xxx.xxx your.domain.name
# By default, we delegate on Azure
fallthrough
}
forward . 168.63.129.16 # This IP is Azure's DNS service
errors
}
EOL
echo "Restarting coredns to pick up new config."
systemctl restart coredns.service
```
- In your client Wireguard configuration, uncomment the DNS server line we left before
- Check that the service is running fine by running `dig google.com`. You should see in the output that your laptop has relied on our new DNS to do the name resolution.
- In the Jumphost, run the following command to disable password based SSH authentication fully. This way, access can only be granted with SSH key pairs, which is way more secure: `sudo sed -i -e 's/#PasswordAuthentication yes/PasswordAuthentication no/g' /etc/ssh/sshd_config; sudo systemctl restart ssh`.
- Remove the AllowSSHInboundTemporarily rule that you added to the NSG `superhog-data-nsg-jumphost-<your-env>`. We don't need that anymore since we can SSH through the VPN tunnel.
- Next, we will deploy a Postgres server to act as the DWH.
- Create a new Azure Database for PostgreSQL flexible servers.
- Basics
- Name it: `superhog-dwh-<your-env>`.
- On field `PostgreSQL version` pick version 16.
- Adapt the sizing to your needs. Only you know how much this server is going to take.
- For field `Authentication method` pick `PostgreSQL authentication only`.
- Name the user admin: `dwh_admin_<your-env>`.
- Give it a password and make sure to note it down.
- Networking
- On field `Connectivity method` select `Private access (VNet Integration)`
- Pick the virtual network `superhog-data-vnet-<your-env>` and the subnet `databases-subnet`.
- Create a new private dns zone. Unfortunately, we can't use `<your-env>.data.superhog.com` for this service.
- Security
- Defaults are fine
- Add tags:
-`team: data`
-`environment: <your-env>`
-`project: dwh`
- Validate the deployment by trying to log into the database with the `dwh_admin_<your-env>` user from your favourite SQL client (you can use DBeaver, for example). Be aware that your VPN connection should be active so that the DWH is reachable from your device.
