counterweight/superhog_personal_notes
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

some stuff

Browse source
This commit is contained in:
Pablo Martin 2024-02-12 18:08:28 +01:00
parent 2e396f4407
commit 68d79977d1
1 changed files with 127 additions and 51 deletions
Download patch file Download diff file Expand all files Collapse all files

166
log.md
View file

@ -1745,8 +1745,6 @@ Direct debits, perhaps finance could use this with lower fees?
- Alex's role once we have a DWH
- He can work great
# 2023-01-24
## Product catchup
@ -1762,8 +1760,6 @@ Direct debits, perhaps finance could use this with lower fees?
- We have a contractor coming along for infrastructure for the next 3 months
- Interview with Jeremy
## Meeting with Leo
- Middleware PBI
@ -1772,24 +1768,19 @@ Direct debits, perhaps finance could use this with lower fees?
## Product KPIs
# 2024-01-26
## Interview with Jeremy
ETL Role - Informatica
Then Support on ETL
Also doing reporting SAS, SAP BO,
Systems Analyst
SQL with MSSQL and Oracle
SSRS
Tableau
- git
- SSRS
- SQL queries
@ -1804,7 +1795,6 @@ Tableau
- He likes quite a bit the DE side
- He prefers the Data Analysis
- Data contracts
- Organizing data consumption
@ -1830,13 +1820,12 @@ Tableau
# 2023-01-29
Today's priorities:
- [ ] Invoicing tool
- [X] Data News
- [X] Invoicing Documentation
- [ ] Wireguard
# 2023-01-30
- [X] Invoicing tool
@ -1856,7 +1845,6 @@ Today's priorities:
-
- Offering and pricing
## Call with Leo
- KYG stuff: please align with Ben C.
@ -1875,9 +1863,10 @@ Today's priorities:
Meeting with Suzanne and Finnian
Suzanne
- Account Manager
- Account Manager
Finnian
- Customer Engineer
- Customer Engineer
- Usual infrastructure pattern
- Integrations with AD for permission handling
@ -1905,13 +1894,13 @@ Finnian
- Scheduled email/slack alerts with conditions on visuals
- Scheduled reports into slack?
We are eligible for startup credits. 100K awarded for 1 year team
Stuff I want to read:;
- https://honeydew.ai/uncategorized/dbt-semantic-layer-vs-looker-lookml/
- https://www.rittmananalytics.com/blog/2022/2/1/lightdash-looker-and-dbt-as-the-bi-tool-metrics-layer/
- https://www.getdbt.com/coalesce-2020/perfect-complements-using-dbt-with-looker-for-effective-data-governance
- <https://honeydew.ai/uncategorized/dbt-semantic-layer-vs-looker-lookml/>
- <https://www.rittmananalytics.com/blog/2022/2/1/lightdash-looker-and-dbt-as-the-bi-tool-metrics-layer/>
- <https://www.getdbt.com/coalesce-2020/perfect-complements-using-dbt-with-looker-for-effective-data-governance>
# 2024-02-01
@ -1925,8 +1914,6 @@ Stuff I want to read:;
- [ ] Document
- [ ]
# 2024-02-02
- [X] Document SH invoicing as Data Product
@ -1936,7 +1923,6 @@ Stuff I want to read:;
- [ ] Set up dbt runner machine in dev
- [ ] Document infra
# 2024-02-05
- Data Catalogue
@ -1945,6 +1931,7 @@ Stuff I want to read:;
- [X] Add invoicing tool as Data Product
Invoicing Reformation
- [X] Finish stage descriptions
- [X] Drop existing knowledge on inputs
- [X] Drop details on Stripe exports
@ -1983,7 +1970,6 @@ Invoicing Reformation
- Organizers for private office <https://www.ikea.com/es/es/p/vattenkar-estante-escritorio-blanco-20541568/> x 4
- Post-its, pens, pencils, a couple boxes of white A4, scissors, tape, etc
- [X] Bitcoin competencies test
- [X] Organize week
- [X] Marcel afternoon
@ -1998,14 +1984,13 @@ Invoicing Reformation
- To be able to ONLY READ from the sync schema
- Nothing else
Strategy:
- Create a `modeler` role.
- The modeler can:
- Read anything in the database
- Do anything in the `staging`, `intermediate` and `reporting` schemas.
```sql
CREATE ROLE modeler WITH NOLOGIN;
@ -2019,9 +2004,8 @@ GRANT ROLE modeler to dbt_user;
```
Testing
```sql
-- Reading from sync schema should work
-- Create on sync schema should not work
@ -2052,7 +2036,6 @@ Testing
- Working on it
- I want to go to production this week
- Louise
- Stripe US account almost ready
- Is that going to take waiver payments? Are we synced with finance? What's the rollout plan?
@ -2065,26 +2048,122 @@ Testing
- [X] Add silly shell script to git repo + instructions on how to deploy
# Now
# 2024-02-07
- [ ] document infra
- Continue
## Product catchup
- Full steam ahead with infrastructure
- Planning on deploying production on friday
- Exploring cosmosdb
- Invoicing
- Moving forward fine.
- Small issues that we will improve, nothing critical.
- I will timebox 4 working days towards the end of the month for improvements and data quality topics
- Data quality and strictness
## Cancellations intro
- Guests can opt-in into cancellation protection
- Guest pays. If he needs to cancel, he does and gets his booking money back.
- We initially wanted to do the nice
- Plays nicely if the host has a super strict cancellation policy, like no
- They will define more reporting needs so we can decide what we implement
- They will loop me in when the database model is being designed so I can provide advice on it
## Invoicing issues
- Several accounts with same hostname, the export files are being overwritten.
- We will find a way to deal with this in next cycle.
# 2024-02-09
## Interview with Barnabas
- git?
- No experience
- python?
- sql/postgres?
- SQL
- Infopool?
- dbt
- reporting/pbi?
- Power BI
- Which tool do you prefer? PBI
- Why?
- "Chemistry"
- Modeling in DWH vs PBI
- Alteryx!
- His example project was a pretty standard ERP thingy...
- "I prefer SQL to Python, but I use python for ad-hoc"
- Data contracts
- Organizing data consumption?
- How do you handle unrealistic expectations?
- Technical vs Business
- What are things that you don't like about the current work and that you would hope would be different?
- How do you keep business people familiar with the reports and metrics you produce?
- Any times you've seen teams failing and why do you think it happened?
- What's the favourite reporting tool?
- Have you experience with ELT?
- 14 years of experience
- SQL
- Python
- PBI/Tableau
- Olympus Portal (show transactions live in POS)
- Specter (self-service learning platform)
- Personalized contextual messaging towards customer for Lloyds
- Tell me more about this
- What was the team like
- What was your exact role
- Migrating from DWH to GCP
- I'll get together with the Ben and Ben and we will be back in touch.
My impression:
- Reading his profile from screen, that wasn't nice.
- Avoided question on technical skills a lot, I'm very unsure about his basic technical skills.
- Very confusing explanations.
- I don't think he can organize company-wide data organization.
# 2024-02-12
- [ ] Last week update
- [X] Send slot to Leo
- [ ] Infra
- [ ] Get service account from Ben R.
## Product weekly
- Got tangled with finance
- Infra is very close.
- Today I'll finish the script
- If invoicing doesn't give me troubles, production should be deployed before thursday
## 2024-02-13
- Infra
- [ ] Make E2E visually with Infratest to check everything is connected properly
- [ ] Tear down infratest resource group
- [ ] Deploy the Production environment
- [ ] Last week update
# Now
- [ ] Plan holidays
Architecture
- [ ] Set up dbt runner properly
- [ ] Document deployment plan
- [ ] Learn how the hell does the Azure cli work with the templates
- [ ] Create repo with automation templates
- [ ] Deploy
Data docs
- [ ] Start working on internal docs for ways of working around DWH, VPN, dbt, etc.
- Data Catalogue
- [ ] Add CosmosDB to data catalogue
@ -2093,13 +2172,10 @@ Data docs
- [ ] Set query parameters in the athena thingy
- [ ] Add the new tab with totals
- [ ] Ask IT for <data-team@superhog.com> mail
- [ ] Place this in Backlog: <https://superhogteam.slack.com/archives/C06A52YUDM1/p1702557374922989>
- [ ] Story about checking for accounts with no deal id
- Data literacy roadmap
- Data consumer
- Anyone who requests it