wip airbyte

human-script.md

@@ -461,6 +461,45 @@ Follow this to deploy the entire data infra.
 
 ## 5. Airbyte
 
+### 5.1 Deploying Airbyte VM
+
+- Airbyte lives on its own VM. To do so, create a new VM following these steps.
+  - Basic settings
+    - Name it: `airbyte-<your-env>`
+    - Use Ubuntu Server 22.04
+    - Use Size: `Standard_DS1_v2`
+    - Use username: `azureuser`
+    - Use the SSH Key: `superhog-data-<your-env>-general-ssh`
+    - Select the option `None` for Public inbound ports.
+  - Disk settings
+    - Increasing the data disk to at least 64gb as a starting point is recommended. Airbyte can be a bit of a disk hog, and running low on space might lead to obscure errors happening. Start with 64gb and monitor as you increase usage.
+  - Networking
+    - Attach to the virtual network `superhog-data-vnet-<your-env>`
+    - Attach to the subnet `services-subnet`
+    - Assign no public IP.
+    - For setting `NIC network security group` select option `None`
+  - Management settings
+    - Defaults are fine.
+  - Monitoring
+    - Defaults are fine.
+  - Advanced
+    - Defaults are fine.
+  - Add tags:
+    - `team: data`
+    - `environment: <your-env>`
+    - `project: airbyte`
+- Once the VM is running, you should be able to ssh into the machine when your VPN is active.
+
+### 5.2 Deploying Airbyte
+
+!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+CONTINUE HERE
+Script out these instructions https://docs.airbyte.com/deploying-airbyte/on-azure-vm-cloud-shell
+
+### 5.3 Test connection to DWH
+
+- This step is optional. If you are deploying production, you should probably skip it to avoid making the DWH dirty.
+
 ## 6. Power BI
 
 ## 7. dbt