- Working out where to change in Terraform to get BigQuery env
- Just managed to set up local env
- Had some issues with nix
- Filling in onboarding details
Onboarding stuff
- I've started and merged this PR to onboarding (on-call): https://github.com/GaloyMoney/onboarding/pull/18
- Started this one, still pending review (code-review): https://github.com/GaloyMoney/onboarding/pull/17
### Chat with Sebastien
From Sebastien
If you haven't figured this out yet, the data flows like this:
staging (stg_* files) -> intermediate (int_* files) -> output (misc. but often report_* files)
Under staging, the 'rollups' folder hosts the "rolled up" source data from the backend, raw.
Under intermediate, the 'rollups' folder hosts the expanded & type casted version of the above raw data and should be the source of all(most?) our transformations
The backend is architecture'd to stream events for most "objects"/"entity"
(as concisely explained here https://www.youtube.com/watch?v=lg6aF5PP4Tc)
and so the rollups are the snapshots of the state of those "objects"/"entity" as a one per row table (chronological reduce / event summarization) mentioned in the video around 4:20.
The reduce process are backend side, done as triggers for now on "objects"/"entity" events table in PG and visible a sql migrations under lana/app/migrations/<date>_*_events_rollup.sqlI think you should be familiar with that process given the interview take home, but that's what we adopted for now and it's all automated...
so if a field name changes in the backend for example, and the pipeline falls out of sync it breaks and we can address it, rather than silently failing.
So anyways all the concepts of the bank and data we might be interested in analyzing, reporting, etc. are implicitly documented with the above 3 set of rollup sql.
The credit facility object is probably the most interesting and easy to start with as it is a vanilla bank loan or dumbed down line of credit.
### Coffee with Kartik
- Where are you based?
- Bangalore, been there for two years
- How long you've been around?
- 5 years, started out when it was just Nicolas and him for engineering?