Commit graph

577 commits

Author SHA1 Message Date
Pablo Martin
4214c6b1c5 schema, small fixes on model 2024-08-21 17:48:06 +02:00
Oriol Roqué Paniagua
85131985d8 Merged PR 2615: Beautification of KPIs dimensions
# Description

Changes:

* Separate 1) the internal naming of dimensions available within DWH vs. 2) the display of the dimensions in the reporting. Mainly it changes the "by_number_of_listings" to display "By # of Listings Booked in 12 Months". I edited the production macro since to me it's linked to when things are available for display.
* Add preceding zeros on the segmentation so it's ordered correctly. Before, the segment 21-60 was displayed before the 6-20.
* Also added some capital letters to the schema config of the reporting model :)

I attach a screenshot of how it looks in PBI in my local development branch to exemplify why this is "Beautification". Be aware that merging this also puts in production the dimensions.

![image.png](https://guardhog.visualstudio.com/4148d95f-4b6d-4205-bcff-e9c8e0d2ca65/_apis/git/repositories/54ac356f-aad7-46d2-b62c-e8c5b3bb8ebf/pullRequests/2615/attachments/image.png)

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #19325
2024-08-21 14:42:05 +00:00
Pablo Martin
40168f74b3 reorder fields 2024-08-21 15:13:19 +02:00
Pablo Martin
34a2279387 some casts 2024-08-21 15:12:05 +02:00
Pablo Martin
64e5d78578 progress in staging, ordered fields 2024-08-21 12:56:45 +02:00
Pablo Martin
2331db2803 add source file 2024-08-21 12:46:05 +02:00
Oriol Roqué Paniagua
83d913f9fc Merged PR 2607: Propagates and exposes multiple dimension handling for KPIs
# Description

This PR ensures the propagation of the dimensions for KPIs across the key aggregating and exposing models. Additionally, provides these 2 new fields in reporting while **not affecting the current data display**, thus it's safe to work in the PBI report without needing to work in 2 PRs in parallel.

**Changes:**

**1 - Intermediate, `int_mtd_vs_previous_year_metrics`:**

* Removes the temporary filter on `where dimension in ({{ production_dimensions }})`. This will be applied directly to reporting later. This ensures that the new dimension on customer segmentation is fully available only within intermediate.
* Adds `dimension` and `dimension_value` granularity. This includes: 1) adding these fields, 2) joining by these fields with all the source CTEs containing the source models with metrics - which in turn needs the change of the dates model - and 3) joining by these fields in the self-join to compute the incremental vs. previous year.
* Changes on the schema file

**2 - Intermediate, `int_mtd_aggregated_metrics`:**

* Adds `dimension` and `dimension_value` granularity. This includes only adding these fields.
* Changes on the schema file

**3 - Reporting, `mtd_aggregated_metrics`:**

* Adds the filter removed on `int_mtd_vs_previous_year_metrics`. This ensures that only the Global dimension is available for the reporting, thus **no changes from user POV**.
* Adds `dimension` and `dimension_value` granularity. This includes only adding these fields
* Changes on the schema file

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #19325
2024-08-20 15:42:27 +00:00
Joaquin Ossa
bf9652f258 Finished adding address_validation to reporting 2024-08-20 14:11:21 +02:00
Joaquin Ossa
db9f9a6c25 Merged PR 2599: Added address_validation to int_core__vr_check_in_cover
# Description

Added address_validation to int_core__vr_check_in_cover for check in hero report in PBI

# Checklist

- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [x] I have checked for DRY opportunities with other models and docs.
- [x] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #17069
2024-08-20 12:06:51 +00:00
Oriol Roqué Paniagua
80abac494a Merged PR 2583: Invoicing metrics per customer segmentation
# Description

Silly change:

* Modifies `int_core__mtd_guest_payments_metrics` to apply the proper key on date, dimension and dimension_value.
* -> *The weird thing is that the previous dbt test I run worked well. Is it possible that the configuration in the schema file prevails on top of the model configuration? I thought it was the other way around...*

Main changes:

* Modifies `int_xero__mtd_invoicing_metrics` to include the customer segmentation based on listings.
* `schema.yaml` is also affected including new fields and tests
* Added the macro to retrieve the production dimension in `int_core__mtd_vs_previous_year_metrics` to avoid propagating this upwards and messing up with the data display.

Overall, follows a similar strategy as we did for Booking, Guest Journey, Deal, Accommodation and Guest Payments metrics. For reference, [here's the previous PR on Guest Payments](https://guardhog.visualstudio.com/Data/_git/data-dwh-dbt-project/pullrequest/2580).

This is the last PR on the source models for KPIs. Will follow: propagation + exposure

# Checklist

- [X] The edited models and dependants run properly with production data.
* **Important note**: this segmentation provides null values for all API-related KPIs. Makes sense, since the 4 deal id we have for APIs do NOT have, or have had, a listing linked to them. I'd say it's not a blocker.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #19325
2024-08-20 12:05:53 +00:00
Oriol Roqué Paniagua
68f490f9fa Merged PR 2593: Modifying int_core__user_host to take into account KYG users
# Description

Adds the possibility of considering as Hosts those users that come from Know Your Guest (KYG), after the discussion with Ben R yesterday. This uses the Claim table, specifically on any Kyg claim type:
- KygRegistrationSignUpType
- KygRegistrationIntegrationTypeName
- KygMvp

From what I see compiling the new vs. the previous version of this model and running into production to have up-to-date data, this increases the number of hosts in 8, from 2.608 to 2.616 so it's not a massive change in volumes.

I also modified the schema for this model to reflect the new logic.

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #19513
2024-08-20 11:27:43 +00:00
Joaquin Ossa
45f7f640ba Changed names for easier understanding 2024-08-20 12:56:52 +02:00
Joaquin Ossa
c9fb99743a Merge remote-tracking branch 'origin/models/17069_addressvalidation_to_reporting' into models/17069_addressvalidation_to_reporting 2024-08-20 12:32:58 +02:00
Joaquin Ossa
8171fda345 Merged PR 2578: Created address_validations in intermediate
# Description

Created address_validations in intermediate for check in hero

# Checklist

- [X] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [x] I have checked for DRY opportunities with other models and docs.
- [x] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #17069
2024-08-20 09:46:27 +00:00
Joaquin Ossa
1c60099b0b Removed coalesce so it includes NULL for failed validations 2024-08-20 10:02:04 +02:00
Joaquin Ossa
798fc7b937 Removed coalesce so it includes NULL for failed validations 2024-08-20 09:49:34 +02:00
Joaquin Ossa
ff962608ad Added Address_validation to int_core__vr_check_in_cover 2024-08-20 09:27:49 +02:00
Oriol Roqué Paniagua
e1d04c2e4e Merged PR 2580: Guest Payments metrics per customer segmentation
# Description

Modifies `int_core__mtd_guest_payments_metrics` to include the customer segmentation based on listings. `schema.yaml` is also affected including new fields, tests and apply the proper naming (from guest revenue to guest payments). I also modified a silly naming that was referring to deals to refer to listings/accommodations, my bad.
Added the macro to retrieve the production dimension in `int_core__mtd_vs_previous_year_metrics` to avoid propagating this upwards and messing up with the data display.

Overall, follows a similar strategy as we did for Booking, Guest Journey, Deal and Accommodation metrics. For reference, [here's the previous PR on Accommodations](https://guardhog.visualstudio.com/Data/_git/data-dwh-dbt-project/pullrequest/2575?_a=overview).

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #19325
2024-08-19 11:45:42 +00:00
Oriol Roqué Paniagua
997cb85c6c Merged PR 2577: Adding get_kpi_dimensions_for_production
# Description

Takes into account @<Pablo Martín> 's feedback from the previous PR, slightly modified. This PR separates 1) the dimensions while developing vs. 2) the dimensions once these are available for production. This are within the same file of macro configuration for KPIs, namely `business_kpis_configuration`.

End-goal, all CTEs in `int_mtd_vs_previous_year_metrics` will read from this new macro `get_kpi_dimensions_for_production`, so eventually we won't need any hardcode once we want to add new dimensions. In the meantime, I'll be adding this new line for each PR (still 2 missing :D)

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [ ] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #19325
2024-08-19 09:57:28 +00:00
Joaquin Ossa
e8b76e130e Created address_validations in intermediate 2024-08-19 11:43:54 +02:00
Oriol Roqué Paniagua
89dd8845cc Merged PR 2575: Accommodation metrics per customer segmentation
# Description

Modifies `int_core__mtd_accommodation_metrics` to include the customer segmentation based on listings. `schema.yaml` is also affected including new fields and tests. Hardcoded `int_core__mtd_vs_previous_year_metrics` to avoid propagating this upwards and messing up with the data display.

Overall, follows a similar strategy as we did for Booking, Guest Journey and Deal metrics. For reference, here's [the previous PR on Deal](https://guardhog.visualstudio.com/Data/_git/data-dwh-dbt-project/pullrequest/2534). I noticed that I mixed the schema tests of Deals and Accommodations, this PR should fix both.

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #19325
2024-08-19 09:03:42 +00:00
Joaquin Ossa
52f42b9e9b Revert "Created address_validation in staging"
This reverts commit 4a6fdd5ac7.
2024-08-16 12:57:54 +02:00
Joaquin Ossa
4a6fdd5ac7 Created address_validation in staging 2024-08-16 12:51:13 +02:00
Joaquin Ossa
63d7ebb6d1 Merged PR 2564: New model for verification_to_address_validation
# Description

New model for verification_to_address_validation for check in hero reporting

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #17069
2024-08-16 09:21:31 +00:00
Joaquin Ossa
be9757b886 Modified source and included test for id_address_validation as unique as well 2024-08-16 11:15:40 +02:00
Joaquin Ossa
1d4a2d7b34 reorganized columns 2024-08-14 17:36:28 +02:00
Joaquin Ossa
1e818ea306 organized columns 2024-08-14 17:34:04 +02:00
Joaquin Ossa
fa5cfb8c66 removed extra _airbyte_raw_id 2024-08-14 17:31:36 +02:00
Joaquin Ossa
fd44f94b78 new verification_to_address_validation model in staging 2024-08-14 17:30:09 +02:00
Joaquin Ossa
cdd3b6a58f new address_validation model in staging 2024-08-14 17:16:35 +02:00
Pablo Martin
2ebf6a711c add tests 2024-08-12 11:20:09 +02:00
Pablo Martin
419f07e77c rename 2024-08-12 11:19:32 +02:00
Pablo Martin
a39f4d24aa add staging model 2024-08-12 11:18:49 +02:00
Pablo Martin
501a063635 add source 2024-08-12 11:16:53 +02:00
Pablo Martin
8c78f880ec change column name, previous was not accurate 2024-08-12 11:08:28 +02:00
Pablo Martin
22ccae5b9a add tests 2024-08-12 11:05:12 +02:00
Pablo Martin
59477a4641 typo 2024-08-12 11:01:09 +02:00
Pablo Martin
51fd0cf3bc add staging model 2024-08-12 11:00:13 +02:00
Pablo Martin
e4b0f54241 add to sources 2024-08-12 10:47:55 +02:00
Pablo Martin
cfe02fb99c add PK test 2024-08-12 10:42:40 +02:00
Pablo Martin
81b900b6d2 add claim to staging 2024-08-12 10:42:29 +02:00
Pablo Martin
0ba3f13055 add claim to sources 2024-08-12 10:37:40 +02:00
Oriol Roqué Paniagua
d323460d1a Merged PR 2534: Deal metrics per customer segmentation
# Description

Modifies `int_core__mtd_deal_metrics` to include the customer segmentation based on listings. `schema.yaml` is also affected including new fields and tests. Hardcoded `int_core__mtd_vs_previous_year_metrics` to avoid propagating this upwards and messing up with the data display.

Overall, follows a similar strategy as we did for Booking and Guest Journey metrics. For reference, [here's the previous PR on GJ](https://guardhog.visualstudio.com/Data/_git/data-dwh-dbt-project/pullrequest/2533).

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #19325
2024-08-09 10:03:35 +00:00
Oriol Roqué Paniagua
6a660aeac8 Merged PR 2533: Guest Journey metrics per customer segmentation
# Description

Modifies `int_core__mtd_guest_journey_metrics` to include the customer segmentation based on listings. `schema.yaml` is also affected including new fields and tests. Hardcoded `int_core__mtd_vs_previous_year_metrics` to avoid propagating this upwards and messing up with the data display.

Overall, follows a similar strategy as we did for Booking metrics.

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #19325
2024-08-09 08:41:14 +00:00
Joaquin Ossa
87bee4a95a Fixed models for mtd_aggregated_metrics 2024-08-08 17:06:11 +02:00
Oriol Roqué Paniagua
6caac0bb02 Merged PR 2527: Revert 'Added relative_increment_with_sign_format'
# Description

_Describe your motivation and changes here._

# Checklist

- [ ] The edited models and dependants run properly with production data.
- [ ] The edited models are sufficiently documented.
- [ ] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

# Description

Added relative_increment_with_sign_format for special formatting in PBI

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Added relative_increment_with_sign_format

Reverts !2524

Related work items: #19559
2024-08-08 13:27:33 +00:00
Oriol Roqué Paniagua
661f9b9245 Merged PR 2526: Revert 'Modified mtd_aggregated_metrics to reporting'
# Description

_Describe your motivation and changes here._

# Checklist

- [ ] The edited models and dependants run properly with production data.
- [ ] The edited models are sufficiently documented.
- [ ] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

# Description

Modified mtd_aggregated_metrics to reporting

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Modified mtd_aggregated_metrics to reporting

Reverts !2525

Related work items: #19559
2024-08-08 13:27:06 +00:00
Joaquin Ossa
0777b3210a Merged PR 2525: Modified mtd_aggregated_metrics to reporting
# Description

Modified mtd_aggregated_metrics to reporting

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Modified mtd_aggregated_metrics to reporting

Related work items: #19559
2024-08-08 13:20:31 +00:00
Oriol Roqué Paniagua
b61fe3f70d Merged PR 2522: Don't Repeat Yourself for KPIs - Applied to Bookings
# Description

What's new:
- Creation of `get_kpi_dimensions`: new macro to have a single point of source of configuration for dimensions for the KPIs. It's a way to enforce global variables on-demand. I kind of like this approach and we could do it for Xero models as well :)
- Modification of `int_core__mtd_booking_metrics` and `int_dates_mtd_by_dimension`: removal of duplicated code within the dimension context. Uses Jinja code and applies different configurations depending on the dimension chosen. Still, different metrics are placed in different CTEs. I believe it might be possible to also configure metrics BUT at the cost of over-complexifying the macro logic, so I wouldn't go for it at this stage.

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] **I have checked for DRY opportunities with other models and docs.**
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #19325
2024-08-08 13:19:57 +00:00
Joaquin Ossa
d11f639562 Modified mtd_aggregated_metrics to reporting 2024-08-08 15:08:11 +02:00