Anaxi

Anaxi is Superhog's tool to perform Extract-Load (EL) syncs between our multiple Cosmos DB databases and our DWH.

How to use the tool

Note: the app has only been used so far in a Linux environment. Windows support is dubious.

Install

• Ensure you have Python 3.10>= and poetry installed.
• Run poetry install to install dependencies.
• Activate the project's virtual environment. You can use poetry shell.
• Test that everything is working by running anaxi smoke-test. You should see a happy pig.

Set up credentials

anaxi needs a few configs and secrets to run.

Regarding Cosmos DB databases: anaxi expects to find a file called cosmos-db.yml in the path ~/.anaxi/cosmos-db.yml. The file should specify one or more Cosmos DB databases, along with the required secrets to interact with them. You can check the example file in this repo named example-cosmos-db.yml to understand how to build this file. Once you've done that, you can check if any database is reachable with the cosmos-db-healthcheck command. See more in the General Usage section below.

For Postgres databases: anaxi expects to find a file called postgres.yml in the path ~/.anaxi/postgres.yml. The file should specify one or more Postgres databases, along with the required secrets to interact with them. You can check the example file in this repo named example-postgres.yml to understand how to build this file. Once you've done that, you can check if any database is reachable with the postgres-healthcheck command. See more in the General Usage section below.

General Usage

You can run a healthcheck against any Cosmos DB database like this:

anaxis cosmos-db-healthcheck --cosmos-db-id <your-db-id>

You can run a healthcheck against Postgres databases like this:

anaxis postgres-healthcheck --postgres-database <your-db-name>

Incremental scopes

• Callable CLI app
• Healthchecks against Cosmos DB doable
• Healthchecks against DWH doable
• Reading from Cosmos DB
• Writing into DWH
• Refactors and improvements

Development

Local Cosmos DB

Microsoft provides tools to run a local emulator of Cosmos DB. The bad news is we have been unable to make it work so far, it always breaks for one reason or another.

You can find instructions here:

• https://learn.microsoft.com/en-us/azure/cosmos-db/emulator
• https://learn.microsoft.com/en-us/azure/cosmos-db/how-to-develop-emulator

What's with the name

Anaxi is short for Anaximander. Anaximander of Miletus was a pre-Socratic Greek philosopher who lived in Miletus. He is often called the "Father of Cosmology" and founder of astronomy.