data-xexe/README.md

# xexe

`xexe` is Superhog's tool to ingest currency rates from xe.com into our DWH. `xexe` is a Python CLI application, and this is the repository where it lives.

## How to use the tool

*Note: the app has only been used so far in a Linux environment. Windows support is dubious.*

### Install

- Ensure you have Python 3.10> and `poetry` installed.
- Run `poetry install` to install dependencies.
- Activate the project's virtual environment. You can use `poetry shell`.
- Test that everything is working by running `xexe smoke-test`. You should see a happy pig.

### Set up credentials

To use `xexe`, you will need to have credentials for the `xe.com` API. Specifically, you need an account id and it's matching api key.

To set up your environment, you should create a `.env` file and place it in `~/.xexe/.env`. You will have to run `xexe` as the right user to ensure the `.env` file is found. You can use the `.env-example` file as a reference. We also recommend running `chmod 400` or `chmod 600` on it for safety.

Once you have done this, you can run `xexe xe-healthcheck`. If the connection to the API was successful, you will see some output telling you so.

### Using

Remember to activate the project virtual environment.

You can use `xexe` to get rates and store them locally like this:

```bash
xexe get-rates --start-date "2024-01-01" --end-date "2024-01-10" --output my_rates.csv
```

You can also run without specifying dates. Not specifying `end-date` will get rates up to today. Not specifying `start-date` will get dates up to last week.

```bash
xexe get-rates --output my_rates.csv
```

`xexe` comes with a set of default currencies, but you can also specify the currencies you want to get data for by passing them like this:

```bash
# Currencies must be valid ISO 4217 codes and be comma-separated
xexe get-rates --currencies USD,EUR,GBP --output my_rates.csv
```

The output file will follow this schema:

- `date`
- `from_currency`
- `to_currency`
- `exchange_rate`
- `exported_at`

The file will contain all the combinations of the different currencies and dates passed. This includes inverse and equal rates.

This is better understood with an example. Find below a real call and its real CSV output:

```bash
xexe get-rates --start-date 2024-01-01 --end-date 2024-01-03 --currencies EUR,USD,GBP --output file.csv --ignore-warnings
```

```csv
from_currency,to_currency,rate,rate_date,exported_at
GBP,EUR,1.15,2024-01-01,2024-06-12T16:24:38
GBP,USD,1.27,2024-01-01,2024-06-12T16:24:38
EUR,USD,1.10,2024-01-01,2024-06-12T16:24:38
GBP,EUR,1.15,2024-01-02,2024-06-12T16:24:38
GBP,USD,1.27,2024-01-02,2024-06-12T16:24:38
EUR,USD,1.10,2024-01-02,2024-06-12T16:24:38
GBP,EUR,1.15,2024-01-03,2024-06-12T16:24:38
GBP,USD,1.26,2024-01-03,2024-06-12T16:24:38
EUR,USD,1.09,2024-01-03,2024-06-12T16:24:38
EUR,GBP,0.87,2024-01-01,2024-06-12T16:24:38
USD,GBP,0.79,2024-01-01,2024-06-12T16:24:38
USD,EUR,0.91,2024-01-01,2024-06-12T16:24:38
EUR,GBP,0.87,2024-01-02,2024-06-12T16:24:38
USD,GBP,0.79,2024-01-02,2024-06-12T16:24:38
USD,EUR,0.91,2024-01-02,2024-06-12T16:24:38
EUR,GBP,0.87,2024-01-03,2024-06-12T16:24:38
USD,GBP,0.79,2024-01-03,2024-06-12T16:24:38
USD,EUR,0.92,2024-01-03,2024-06-12T16:24:38
GBP,GBP,1,2024-01-01,2024-06-12T16:24:38
USD,USD,1,2024-01-01,2024-06-12T16:24:38
EUR,EUR,1,2024-01-01,2024-06-12T16:24:38
GBP,GBP,1,2024-01-03,2024-06-12T16:24:38
USD,USD,1,2024-01-03,2024-06-12T16:24:38
EUR,EUR,1,2024-01-03,2024-06-12T16:24:38
GBP,GBP,1,2024-01-02,2024-06-12T16:24:38
USD,USD,1,2024-01-02,2024-06-12T16:24:38
EUR,EUR,1,2024-01-02,2024-06-12T16:24:38
```

A few more details:

- Running `get-rates` with an `end-date` beyond the current date will ignore the future dates. The run will behave as if you had specified today as the `end-date`.
- Trying to place an `end-date` before a `start-date` will cause an exception.
- Running with the option `--dry-run` will run against a mock of the xe.com API. Format will be valid, but all rates will be fixed. This is for testing purposes.

## Testing

This CLI app has three groups of tests:

- `tests_cli`: simulate calling the CLI to check proper calls, not much attention paid to actual results.
- `tests_unit`: unit tests for some domain-heavy parts of the codebase.
- `tests_integration`: full executions that assert the end result of the calls to be as expected.

You can run everything with `pytests tests`, or narrow it down more if you want to.

There is a special test in `tests_integration` that runs against the real xe.com API. This tests is commented out to avoid repeteadly consuming API hits. You can use by uncommenting it manually. I know it's annoying, but then again, it shouldn't to be very annoying since you should only use that test sparingly.