diff --git a/log.md b/log.md index 1501873..63ffbc6 100644 --- a/log.md +++ b/log.md @@ -1916,44 +1916,36 @@ Stuff I want to read:; # 2024-02-01 - [X] Invoicing -- [ ] Ryan -- [ ] Leo -- [ ] Invoicing backlog +- [X] Ryan +- [X] Leo +- [X] Invoicing headsup to Stakeholders +- [X] Invoicing backlog - [ ] Wireguard + - [ ] Create interface for all traffic + - [ ] Document + - [ ] -# Now +# 2024-02-02 + +- [X] Document SH invoicing as Data Product +- [X] Document Stripe reports as Data Product +- [X] DOcument Stripe as a data source +- [X] Try out Stripe integration with Airbyte +- [ ] Set up dbt runner machine in dev +- [ ] Document infra -Wireguard VPN -- Delete everything -- Set up firewall listening on 52420 -- Create point to point - - Set it up - - Hit dummy HTTP from Windows - - Hit dummy HTTP from Linux - - Hit coredns from Windows - - Hit coredns from Linux - - Unsuccessfully try to jump anywhere else -- Extend to point to site - - Change server config - - Try to reach airbyte web from windows - - Try to reach airbyte web from linux - - Try to reach airbyte ssh from windows - - Try to reach airbyte ssh from linux - - Try to reach DWH from windows - - Try to reach DWH from linux - - Try to reach Core from windows - - Try to reach Core from linux - +# 2024-02-05 -Architecture -- [ ] Document dev ramblings +- Data Catalogue + - [X] Make big announcement + Quick Presentation Video + - [X] Add Stripe reporting as data product in data catalogue + - [X] Add invoicing tool as Data Product Invoicing Reformation - -- [ ] Finish stage descriptions +- [X] Finish stage descriptions - [X] Drop existing knowledge on inputs - [X] Drop details on Stripe exports - [X] Get access to Stripe from Louise @@ -1967,7 +1959,7 @@ Invoicing Reformation - [X] Backlog discussion - [X] Quick coffee with Amanda - [X] Phase 1 summary in Notion + Email -- [ ] January processing +- [X] January processing - New improvements - Automated Stripe Transaction Exports - Exchange Rate Formalization @@ -1985,27 +1977,129 @@ Invoicing Reformation - Pablo runs the exports - Everything gets shared with Finance team -- Data Catalogue - - [X] Make big announcement + Quick Presentation Video - - [ ] Add CosmosDB to data catalogue - - [ ] Add Stripe reporting as data product in data catalogue - - [ ] Add invoicing tool as Data Product - - - -- [ ] Ask IT for mail -- [ ] Place this in Backlog: -- [ ] Add story to backlog about rough DealId compliance monitoring through quick and dirty weekly extract -- [ ] Story about checking for accounts with no deal id -- [ ] Story about report to spot non-matching pricing details across Dashboard Superhog. -- [ ] Story about having a currency exchange rate database in DWH - -- [ ] Equipment +- [X] Equipment - Bookcase for private office x 1 - Organizers for private office x 4 - Organizers for private office x 4 - Post-its, pens, pencils, a couple boxes of white A4, scissors, tape, etc + +- [X] Bitcoin competencies test +- [X] Organize week +- [X] Marcel afternoon +- [X] Schedule with candidate +- [X] sh-invoicing issue +- [X] dbt runner + +## Permissions for dbt user in DWH + +- It needs + - To be able to create schemas and operate on them + - To be able to ONLY READ from the sync schema + - Nothing else + + +Strategy: +- Create a `modeler` role. +- The modeler can: + - Read anything in the database + - Do anything in the `staging`, `intermediate` and `reporting` schemas. + + +```sql + +CREATE ROLE modeler WITH NOLOGIN; +ALTER SCHEMA staging OWNER TO modeler; +ALTER SCHEMA intermediate OWNER TO modeler; +ALTER SCHEMA reporting OWNER TO modeler; + +CREATE ROLE dbt_user WITH PASSWORD ; +GRANT ROLE modeler to dbt_user; + + +``` + + + +Testing +```sql +-- Reading from sync schema should work +-- Create on sync schema should not work +-- Inserting on sync schema should not work + +-- Creating schema should work +-- Creating new table in new schema should work +-- Insert data in new table in news chema should work +-- Reading from that new table should work +-- Creating a view in new schema should work + +-- Creating new table in schema `staging` should work +-- Inserting data in new table in newschema should work +-- Reading from the new table in `staging` should work +-- Creating a view in `staging` should work + +-- Doing anything on `public schema should not work` + +``` + +## Product weekly + +- Finance + - They are using things smoothly, success + - Also collecting small details + - Any updates from other teams on this? +- Infra + - Working on it + - I want to go to production this week + + +- Louise + - Stripe US account almost ready + - Is that going to take waiver payments? Are we synced with finance? What's the rollout plan? + - Yes, all payments in US dollar will go through there from Monday onwards. + - I will receive access to the Stripe US account. + - Who's doing this? + - Louise, but she will hand things over to Amanda. + +## 2024-02-06 + +- [X] Add silly shell script to git repo + instructions on how to deploy + +# Now + +- [ ] document infra + - Continue + +- [ ] Plan holidays + + +Architecture +- [ ] Set up dbt runner properly +- [ ] Document deployment plan +- [ ] Learn how the hell does the Azure cli work with the templates +- [ ] Create repo with automation templates +- [ ] Deploy + +Data docs +- [ ] Start working on internal docs for ways of working around DWH, VPN, dbt, etc. + + + +- Data Catalogue + - [ ] Add CosmosDB to data catalogue + +- #13121 + - [ ] Get access from Ray + - [ ] Set query parameters in the athena thingy + - [ ] Add the new tab with totals + + +- [ ] Ask IT for mail +- [ ] Place this in Backlog: +- [ ] Story about checking for accounts with no deal id + + + - Data literacy roadmap - Data consumer - Anyone who requests it