final touches from real life testing
This commit is contained in:
parent
1996bab595
commit
4035d96369
1 changed files with 24 additions and 7 deletions
|
|
@ -854,7 +854,7 @@ echo "🔍 Verifying Docker and Docker Compose versions..."
|
|||
docker --version
|
||||
docker-compose --version
|
||||
|
||||
usermod -a -G docker $USER
|
||||
sudo usermod -a -G docker $USER
|
||||
newgrp docker
|
||||
|
||||
echo "✅ Docker and Docker Compose installation completed!"
|
||||
|
|
@ -876,25 +876,42 @@ sudo apt-get update
|
|||
sudo apt-get install -y postgresql-client-16
|
||||
```
|
||||
|
||||
#### 080.1.4 Create user in DWH
|
||||
#### 080.1.4 Install Python
|
||||
|
||||
- Python is needed to create virtual environments and run dbt and other commands.
|
||||
- You can use the following script to install python and some required packages:
|
||||
|
||||
```bash
|
||||
sudo apt-get install python3.10 python3.10-venv
|
||||
```
|
||||
|
||||
#### 080.1.5 Create user in DWH
|
||||
|
||||
- The CI Postgres will use some Foreign Data Wrappers (FDW) pointing at the DWH. We will need a dedicated user in the DWH instance to control the permissions received by the CI server.
|
||||
- The section of this guide dedicated to setting up the DWH explains how to create this user. If you have followed it, you might have already created the user. Otherwise, head there to complete this part.
|
||||
|
||||
#### 080.1.5 Install the Azure Devops agent and sync with Devops
|
||||
#### 080.1.6 Install the Azure Devops agent and sync with Devops
|
||||
|
||||
- The VM needs to have a Microsoft provided Azure agent to be reachable by Devops. This agent listens to requests from Devops, basically allowing Devops to execute things on the VM.
|
||||
- Some configuration needs to be done in the Azure Devops project to allow Azure Devops to reach the VM.
|
||||
- Some configuration needs to be done in the Azure Devops project to allow Azure Devops to reach the VM. This might include creating a pool if it doesn't exist.
|
||||
- You can find how to set this up in Ubuntu in these links:
|
||||
- Official MSFT docs: https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/linux-agent?view=azure-devops
|
||||
- Helpful walkthrough video: https://www.youtube.com/watch?v=Hy6fne9oQJM
|
||||
- Make sure to install the agent as a systemd service to have it always run on boot. The details are explained in Microsoft's documentation.
|
||||
- Once the agent is installed and correctly linked to one of our Pools in Devops, you should see the agent listed in the Devops UI for that pool, with status online. Don't move on if you haven't succeeded on this point.
|
||||
|
||||
#### 080.1.6 Clone the project and further steps
|
||||
#### 080.1.7 Clone the project and further steps
|
||||
|
||||
- We are going to need a local clone of the git repository to perform some setup steps, as well as for business as usual execution.
|
||||
- To do this:
|
||||
- Use or create some SSH key to have access to clone repos from Azure Devops. This could be the key `superhog-data-general-ssh-<your-env>` or some other key. This guide leaves this detail up to you. You can read more on how to use SSH keys with Azure Devops here: https://learn.microsoft.com/en-us/azure/devops/repos/git/use-ssh-keys-to-authenticate?view=azure-devops.
|
||||
- Once the CI VM is capable, clone the dbt project into the `azureuser` home dir.
|
||||
- Add some SSH key to the VM to have access to clone repos from Azure Devops. This could be the key `superhog-data-general-ssh-<your-env>` or some other key. This guide leaves this detail up to you. You can read more on how to use SSH keys with Azure Devops here: https://learn.microsoft.com/en-us/azure/devops/repos/git/use-ssh-keys-to-authenticate?view=azure-devops.
|
||||
- Also add this config to make SSH cloning work. Note that the details might have changed since this guide was written, so your mileage may vary.
|
||||
```
|
||||
Host ssh.dev.azure.com
|
||||
Hostname ssh.dev.azure.com
|
||||
IdentityFile ~/.ssh/<whatever-key-file-you-are-using>
|
||||
```
|
||||
- Once the CI VM is SSH capable, clone the dbt project into the `azureuser` home dir.
|
||||
- There are several steps after this, for which you should find instructions in the dbt repository itself.
|
||||
|
||||
## 090. Monitoring
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue