working

README.md

@@ -6,10 +6,10 @@ Content is structured as follows:
 
 - `human-script.md` is a deployment script for you to read and follow. It guides you through all actions you should take.
 - `architecture-overview.md` is an overview of the final architecture that results of following the humna script. If you are not familiar with our architecture, it probably makes sense to read this first so you know what you are setting up.
-- `./templates` contains a set of Azure templates to deploy the different services in a subscription. The `human-script.md` will guide you in when you should run what.
 - `monitoring-and-administration.md` contains guidelines on how to keep the lights on on several of the components.
 
 The following contents are not covered in this repository:
 
 - Application-level configuration for DWH, Airbyte, Power BI, dbt, etc.
 - Instance types, disk sizes and other elements will come with defaults. But these are values that you should adapt to the deployment according to volumen and needs, and probably will also change with time. Treat the defaults with skepticism and make sure you adapt to your needs.
+- No IaaC definitions exist as of today. This guide expects you to deploy things manually.

human-script.md

@@ -2,14 +2,128 @@
 
 Follow this to deploy the entire data infra.
 
-## 0. Pre-requisites
+## 0. Pre-requisites and conventios
 
 - You need an Azure subscription and a user with administrator rights in it.
+- Whenever you see `<your-env>`, you should replace that with `dev`,`uat`, `prd` or whatever fits your environment.
+- We traditionally deploy resources on the `UK South` region. Unless stated otherwise, you should deploy resources there.
 
 ## 1. Resource group
 
+- Create a resource group. This resource group will hold all the resources. For the rest of this guide, assume this is the resource group where you must create resources.
+- Name it: `superhog-data-rg-<your-env>`
+- Add tags:
+  - `team: data`
+  - `environment: <your-env>`
+
 ## 2. Networking
 
+### 2.1 VNET
+
+- Create a virtual network. This virtual network is where all our infra will live. For the rest of this guide, assume this is the network where you must connect services.
+- Name it: `superhog-data-vnet-<your-env>`
+- You need to think what the network range should be like. For example, you could decide that the entire vnet will be contained within. For reference, we should be fine with a `/24` space (256 addresses) since we will only have a handful network interfaces connecting.
+- As an example, we will use `10.69.0.0/24`. This link might be helpful: <https://www.davidc.net/sites/default/subnets/subnets.html?network=10.69.0.0&mask=24&division=11.f10>
+- You need to add three subnets:
+  - Add no network security groups to any of the subnets still. We will create those later.
+  - Jumphost subnet
+    - This subnet is where jumphost boxes will live.
+    - It will be the only subnet where we allow inbound connections from WAN.
+    - Name it `jumphost-subnet`.
+    - For our example, we will make it `10.69.0.0/29` (8 addresses).
+  - Database subnet
+    - This subnet is where the DWH database will live.
+    - Inbound traffic will be allowed from both the jumphost subnet as well as the services subnet.
+    - Name it `database-subnet`
+    - For our example, we will make it `10.69.0.8/29` (8 addresses).
+  - Services subnet
+    - This subnet is where most VMs dedicated to data services live (Airbyte, dbt, PBI Data Gateway, etc).
+    - Inbound traffic will only be allowed from the jumphost subnet.
+    - Name it `services-subnet`
+    - For our example, we will make it `10.69.0.64/26` (64 addresses)
+- Add tags:
+  - `team: data`
+  - `environment: <your-env>`
+  - `project: network`
+
+### 2.2 Network security groups
+
+- You will create three network security groups (NSG)
+- Jumphost NSG
+  - Name it: `superhog-data-nsg-jumphost-<your-env>`
+  - Purpose: only allow connecting to the VPN server. We deny absolutely any other inbound traffic.
+  - Add tags:
+    - `team: data`
+    - `environment: <your-env>`
+    - `project: network`
+  - Add the following inbound rules
+    - VPN Rule
+      - Name: AllowWireguardInbound
+      - Source: Any
+      - Source port ranges: *
+      - Destination: the addresss range for the `jumphost-subnet`. In this example, `10.69.0.0/29`.
+      - Destination port ranges: 51420
+      - Protocol: UDP
+      - Action: Allow
+      - Priority: 100
+    - Deny Rule
+      - Name: DenyAllInbound
+      - Source: Any
+      - Source port ranges: *
+      - Destination: Any
+      - Destination port ranges: *
+      - Protocol: Any
+      - Action: Allow
+      - Priority: 1000
+- Services NSG
+  - Name it: `superhog-data-nsg-services-<your-env>`
+  - Purpose: only allow the service VMs to be reached from our jumphost subnet. We deny absolutely any other inbound traffic.
+  - Add tags:
+    - `team: data`
+    - `environment: <your-env>`
+    - `project: network`
+  - Add the following inbound rules
+    - SSH Rule
+      - Name: AllowSSHFromJumphostInbound
+      - Source: the addresss range for the `jumphost-subnet`. In this example, `10.69.0.0/29`.
+      - Source port ranges: *
+      - Destination: the addresss range for the `services-subnet`. In this example, `10.69.0.64/26`.
+      - Destination port ranges: 22
+      - Protocol: TCP
+      - Action: Allow
+      - Priority: 100
+    - SSH Rule
+      - Name: AllowRDPFromJumphostInbound
+      - Source: the addresss range for the `jumphost-subnet`. In this example, `10.69.0.0/29`.
+      - Source port ranges: *
+      - Destination: the addresss range for the `services-subnet`. In this example, `10.69.0.64/26`.
+      - Destination port ranges: 3389
+      - Protocol: TCP
+      - Action: Allow
+      - Priority: 110
+    - Airbyte web rule
+      - Name: AllowAirbyteWebFromJumphostInbound
+      - Source: the addresss range for the `jumphost-subnet`. In this example, `10.69.0.0/29`.
+      - Source port ranges: *
+      - Destination: the addresss range for the `services-subnet`. In this example, `10.69.0.64/26`.
+      - Destination port ranges: 80
+      - Protocol: TCP
+      - Action: Allow
+      - Priority: 120
+    - Deny Rule
+      - Name: DenyAllInbound
+      - Source: Any
+      - Source port ranges: *
+      - Destination: Any
+      - Destination port ranges: *
+      - Protocol: Any
+      - Action: Allow
+      - Priority: 1000
+  - Add tags:
+    - `team: data`
+    - `environment: <your-env>`
+    - `project: network`
+
 ## 3. Jumphost
 
 ## 4. DWH