Commit graph

811 commits

Author SHA1 Message Date
Pablo Martín
02520d87ee Merged PR 2858: Split Booking Metrics
# Description

This is a pure refactor PR for optimization purposes.

The PR splits the model `int_core__mtd_booking_metrics` into four different models, one for each of the metrics that were computed in the old model.

The only model that depended on `int_core__mtd_booking_metrics` was `int_mtd_vs_previous_year_metrics`. Now it depends on the four new models instead.

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #21438
2024-09-16 14:34:42 +00:00
Joaquin Ossa
3d8ede273e Simplified it 2024-09-16 16:20:36 +02:00
Joaquin Ossa
54af8af30e Update mtd_aggregated_metrics 2024-09-16 16:01:49 +02:00
Pablo Martin
aeb4538e50 remove dangling comment 2024-09-16 14:45:09 +02:00
Pablo Martin
fda6647637 update schemas 2024-09-16 14:40:01 +02:00
Pablo Martin
385c49bf2c remove old booking metrics 2024-09-16 14:35:12 +02:00
Pablo Martin
aa06b28a69 refactor booking metrics children to use new models 2024-09-16 13:02:42 +02:00
Pablo Martin
4140f5b917 create individual models for each booking metric 2024-09-16 13:02:09 +02:00
Joaquin Ossa
2c17df3742 Merged PR 2854: Exposure for e-deposit invoice
# Description

Exposure for e-deposit invoice

# Checklist

- [ ] The edited models and dependants run properly with production data.
- [ ] The edited models are sufficiently documented.
- [ ] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Exposure for e-deposit invoice

Related work items: #20124
2024-09-16 10:25:19 +00:00
Oriol Roqué Paniagua
c1b97e17e6 Merged PR 2852: Fix: ensure priority selection on user migration
# Description

Fixing logic to ensure priority selection of claims when user satisfies multiple claim conditions.
It adds a new parameter that forcefully prioritises the selection of the date value for a certain claim over the others. If the value is repeated among claims, it will select the earliest date.

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #20773
2024-09-16 09:34:44 +00:00
Joaquin Ossa
8fd5061989 Exposure for e-deposit invoice 2024-09-16 11:14:43 +02:00
Oriol Roqué Paniagua
cf1d6e28cc Merged PR 2849: Fix: KYG lite migration with proper date migration handling
# Description

This PR fixes the New Dash migration issue that happened on September 10th 2024. In this migration, users were directly assigned the claim of KygMvp that does not contain a date value. We were using a default hardcode of the first MVP migration, thus in DWH all users have been considered to be migrated late July instead of splitting the first 22 in late July and the ~200 others in September.

The issue lies in the fact that users have configured a ProductBundle and can have Bookings with ProductBundle BEFORE the migration date, which greatly breaks the logic of a migration monitoring.

Changes:
* New migration phase added based on the claim MvpMigratedUser, that Ben created on Friday 13th
* Adaptation of the code in int_core__user_migration to detect if the claim_value (a text field) has a date or not. If so, use that date as long as it's equal or greater than the deployment date, if not use the deployment date. If the claim does not contain a date, use the deployment date (this is the case for the first true 22 migrated users)

I checked that volumes now look correct with this fix.

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #20773
2024-09-16 07:57:41 +00:00
Joaquin Ossa
80a6b225b3 Merged PR 2846: removing duplicates from guesty and removing tests for edeposit_users
# Description

removing duplicates from guesty and removing tests for edeposit_users and edeposit_verification_fees until further notice

# Checklist

- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [x] I have checked for DRY opportunities with other models and docs.
- [x] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

removing duplicates from guesty and removing tests for edeposit_users

Related work items: #20125
2024-09-13 15:16:08 +00:00
Joaquin Ossa
d8501e2df7 kept first updated record instead of last 2024-09-13 17:14:34 +02:00
Joaquin Ossa
32c8ffb655 removef all tests for edeposit_verification_fees 2024-09-13 16:31:47 +02:00
Joaquin Ossa
5551c97db0 removing duplicates from guesty and removing tests for edeposit_users 2024-09-13 16:14:03 +02:00
Pablo Martín
a8b4c08328 Merged PR 2771: Compute Taxes in intermediate
# Description

This model joins a couple of models to `int_core__verification_payments` and adds a few fields around taxs (amounts with and without taxes, tax rates, etc).

## __ATTENTION!!!__

This PR uses `dbt`'s [Model Versioning](https://docs.getdbt.com/docs/collaborate/govern/model-versions) and [Deprecation](https://docs.getdbt.com/reference/resource-properties/deprecation_date) features for the first time. Notice the extra, non-usual fluff in the `schema.yml` file, along with the fact that some model files are suffixed with versions.

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
    - _Note that there are a couple of tests that are deactivated (commented out) due to bad data in the backend. Engineering team is working on fixiing that data. I've decided to comment this for now so we can move forward, but I won't be marking the story on this done until the tests are live in production._
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #20043
2024-09-13 13:24:25 +00:00
Pablo Martin
6434c5c88d replace custom generic test with standard range between one 2024-09-13 15:21:22 +02:00
Pablo Martin
4cf74db49a replace hardcode with variable 2024-09-13 15:21:22 +02:00
Pablo Martin
092b8bd725 clarify refund is not in GBP 2024-09-13 15:21:22 +02:00
Pablo Martin
ec8490527d reorder case for readability 2024-09-13 15:20:39 +02:00
Pablo Martin
0312a1ccd8 remove explicit version reference, rely on latest 2024-09-13 15:20:39 +02:00
Pablo Martin
da3070949a finish columns and schema 2024-09-13 15:20:39 +02:00
Pablo Martin
63599d7a9b add columns 2024-09-13 15:20:39 +02:00
Pablo Martin
15139e00d0 percentage test 2024-09-13 15:20:39 +02:00
Pablo Martin
8c96b96ee5 small comment 2024-09-13 15:20:39 +02:00
Pablo Martin
6136db8403 cte for pre-defining if vat is applicable or not 2024-09-13 15:20:39 +02:00
Pablo Martin
f58e97a3f2 set references for v1 2024-09-13 15:20:39 +02:00
Pablo Martin
2cd115eede remove unnecessary alias, remove unrelated errors in yaml 2024-09-13 15:20:39 +02:00
Pablo Martin
1974da99f7 fix typos 2024-09-13 15:20:39 +02:00
Pablo Martin
d4b797c741 add alias 2024-09-13 15:20:39 +02:00
Pablo Martin
4efc1ba50f v1 goes back to original state 2024-09-13 15:20:39 +02:00
Pablo Martin
bbb9558f62 yaml for new version, plus deprecation 2024-09-13 15:20:39 +02:00
Pablo Martin
4ba6c80d6f v2 model 2024-09-13 15:20:15 +02:00
Pablo Martin
bde6f12404 add new columns 2024-09-13 15:20:15 +02:00
Pablo Martin
0aaec6a619 fuck around with Uri 2024-09-13 15:20:15 +02:00
Pablo Martín
ac1aed029d Merged PR 2832: Format YAML on save + complements
# Description

This PR:
- Adds some `vscode` settings that make VSCode autoformat yaml files on save, like we do with `.sql` files. This should help with formatting consistency.
- I also made some notes about this on the readme.
- Finally, I made a couple of tiny unrelated changes on the readme since I was there.

Related work items: #21344
2024-09-13 08:33:26 +00:00
Pablo Martin
1043c9e80b final files 2024-09-12 15:41:34 +02:00
Pablo Martin
df16cdb00c reporting schema 2024-09-12 15:39:49 +02:00
Pablo Martin
05d5cc6d10 fix schemas in intermediate 2024-09-12 15:38:50 +02:00
Pablo Martin
aabb45dbd5 format files in staging 2024-09-12 15:36:15 +02:00
Pablo Martin
8fdb88afc7 update readme 2024-09-12 15:35:13 +02:00
Pablo Martin
51275b9513 add autoformat on save 2024-09-12 15:25:45 +02:00
Oriol Roqué Paniagua
2295679fc9 Merged PR 2828: Exposing Deal info to reporting
# Description

Exposes the 2 new deal fields to reporting to make it accessible for PBI

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Exposing Deal info to reporting

Related work items: #18911, #19083
2024-09-12 12:29:37 +00:00
uri
0e94022544 Exposing Deal info to reporting 2024-09-12 14:08:36 +02:00
Oriol Roqué Paniagua
c336081d3d Merged PR 2825: Propagates deal Name and Billing Country in int_monthly_aggregated_metrics_history_by_deal
# Description

Changes (only in intermediate):
* Applies sqlfmt in KPIs source models (for some of them it was already applied). Specifically, the 3 Core models ONLY contains formatting changes
![image.png](https://guardhog.visualstudio.com/4148d95f-4b6d-4205-bcff-e9c8e0d2ca65/_apis/git/repositories/54ac356f-aad7-46d2-b62c-e8c5b3bb8ebf/pullRequests/2825/attachments/image.png)

* Adds `main_deal_name` and `main_billing_country_iso_3_per_deal` in `int_monthly_aggregated_metrics_history_by_deal`

* Adds the 2 new fields in the schema entry of `int_monthly_aggregated_metrics_history_by_deal`, including the dbt test not null in the deal name.

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #18911, #19083
2024-09-12 12:04:04 +00:00
Oriol Roqué Paniagua
72c966631e Merged PR 2824: Propagates Billing Country and Deal Name into int_dates_by_deal
# Description

Changes:
* Adding `main_deal_name` and `main_billing_country_iso_3_per_deal` in `int_dates_by_deal` model.
* Documents the 2 new fields. Also, ensures `main_deal_name` is not null
* Removes `id_deal not null` condition since it's enforced on the inner join with `int_core__deal`

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #18911, #19083
2024-09-12 10:27:56 +00:00
Joaquin Ossa
83d98c59b1 Merged PR 2796: Fixed model that wasn't mapping correctly
# Description

Fixed model that wasn't mapping correctly because I am dumb

# Checklist

- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [x] I have checked for DRY opportunities with other models and docs.
- [x] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Fixed model that wasn't mapping correctly

Related work items: #20125
2024-09-10 10:30:08 +00:00
Joaquin Ossa
91296fa91c Fixed model that wasn't mapping correctly 2024-09-10 12:09:16 +02:00
Joaquin Ossa
d5151e2e90 Merged PR 2788: Update edeposit users table
# Description

Created a mapping for some users that have a different id in Cosmos so we can join them using data that Ray shared with me, with this we can get all currency information for each user

# Checklist

- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [x] I have checked for DRY opportunities with other models and docs.
- [x] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #20125
2024-09-10 08:35:37 +00:00