counterweight/superhog-infra-repo
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

rename file

Browse source
This commit is contained in:
Pablo Martin 2024-02-21 15:43:59 +01:00
parent 72a99cf315
commit fd86da85bf
1 changed files with 6 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

13
architecture-overview.md → platform-overview.md
View file

@ -1,8 +1,8 @@
# Architecture Overview # Platform Overview
Our infrastructure is designed to run on Azure. Our infrastructure is designed to run on Azure.
The data infra architecture provides the following services: The data platform provides the following services:
- A PostgreSQL server which acts as a DWH. - A PostgreSQL server which acts as a DWH.
- A self-hosted Airbyte service that acts as a data integration tool (E and L out of ELT). - A self-hosted Airbyte service that acts as a data integration tool (E and L out of ELT).
@ -11,19 +11,18 @@ The data infra architecture provides the following services:
- A simple scheduled dbt run for a dbt project that runs on top of the DWH. - A simple scheduled dbt run for a dbt project that runs on top of the DWH.
- A VPN Server + DNS Resolution to allow developers and power users to access the different services. - A VPN Server + DNS Resolution to allow developers and power users to access the different services.
The infra serves Superhog in the following way: The platform serves Superhog in the following way:
- Data gets ingested from several sources into our DWH. - Data gets ingested from several sources into our DWH. Typically with Airbyte, but other options might be needed for specific cases.
- We perform data cleaning and modeling inside the DWH with dbt. This results in tables in a reporting schema that support our data needs. - We perform data cleaning and modeling inside the DWH with dbt. This results in tables in a reporting schema that support our data needs.
- Data team members and power users build PBI reports and other data products on top of the reporting schema. - Data team members and power users build PBI reports and other data products on top of the reporting schema.
- Data team members and other analysts can also rely on direct access to the DWH to perform ad-hoc analysis and basically cover any data needs that go beyond PBI reports. - Data team members and other analysts can also rely on direct access to the DWH to perform ad-hoc analysis and basically cover any data needs that go beyond PBI reports.
The data infra relies on the following main components: The data infra relies on the following main Azure components:
- A subscription to hold everything. - A subscription to hold everything.
- A resource group to hold all resources. - A resource group to hold all resources.
- A private network. - A private network and three subnets.
- Three subnets.
- A private DNS zone. - A private DNS zone.
- A managed PostgreSQL server. - A managed PostgreSQL server.
- A handful of VMs to host services. - A handful of VMs to host services.