# Description
fixed testing for core__check_in_cover_listings and removed null in accepted values for int_core__bookings in verification_request_booking_source
# Checklist
- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [x] I have checked for DRY opportunities with other models and docs.
- [x] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #20229
# Description
_Describe your motivation and changes here._
# Checklist
- [ ] The edited models and dependants run properly with production data.
- [ ] The edited models are sufficiently documented.
- [ ] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
# Description
Automates the new dash query through DWH. It's a first step towards enabling a very simple PBI reporting on new dash performance. Data is aggregated at user level with the main performance indicators that we're currently reporting. I added a few more informative fields from user side.
Changes:
* in intermediate, creation of `int_core__new_dash_user_overview`. It mainly uses the models in previous PRs to avoid crazy computations, since most of the logic has already been implemented. The only thing is that it's "hardcoded" (with variable) the categorisation of what is NOT a paid service.
* in reporting, creation of `core__new_dash_user_overview`. It just exposes the info from intermediate to reporting.
* added schema entries for intermediate/core and reporting/core, it's the same description for both cases.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Reverts !2657
Related work items: #19570
# Description
Automates the new dash query through DWH. It's a first step towards enabling a very simple PBI reporting on new dash performance. Data is aggregated at user level with the main performance indicators that we're currently reporting. I added a few more informative fields from user side.
Changes:
* in intermediate, creation of `int_core__new_dash_user_overview`. It mainly uses the models in previous PRs to avoid crazy computations, since most of the logic has already been implemented. The only thing is that it's "hardcoded" (with variable) the categorisation of what is NOT a paid service.
* in reporting, creation of `core__new_dash_user_overview`. It just exposes the info from intermediate to reporting.
* added schema entries for intermediate/core and reporting/core, it's the same description for both cases.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19570
# Description
Adds accommodation to product bundle in intermediate. The source table is currently empty, because there's no product bundle different to Basic Screening applied into a listing yet.
Thus, it's possible that this modelisation needs to change in the future since not having data forces us to trust the business logic without a proper technical documentation.
# Checklist
- [ ] The edited models and dependants run properly with production data. **N/A no data**
- [X] The edited models are sufficiently documented.
- [ ] The edited models contain PK tests, and I've ran and passed them. **N/A no data**
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models. **N/A no data - but view should make the trick for the time being**
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19570
# Description
Fixed bookings test to include null for verification_request_booking_source
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #20231
# Description
Creates a new view with the bookings that come from New Dash, linked to the Product Bundles. Extensive documentation added, please check it out and if something is not clear I'll modify it.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models. To be discussed. There's not huge data still so view looks ok to me. I guess, maybe, in the future, we will have a flag or similar in int_core__bookings to detect if a bookings comes from New Dash/New Pricing or not, but it's still early to decide. In any case this model is needed for immediate New Dash MVP reporting so I propose to keep it like this as a view and afterwards we can decide if it makes more sense to materialise it differently.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19570
# Description
Fixes int_dates_by_deal tests
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #20318, #20319
# Description
Adds three more columns to the staging model of `verifications` for edeposit: `ListingId`, `NightlyFee` and `CompanyName`.
# Checklist
- [X] The edited models and dependants run properly with production data.
- ~~[ ] The edited models are sufficiently documented.~~ __No, but I promise they will be.__
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #20123
# Description
Adds accommodation to product bundle table from sync_core to staging. Note that this table still has no data because so far no listing has an associated product bundle.
Small change: removed in schema pending confirmation comments from Lou now that we've got an answer confirming it.
# Checklist
- [ ] The edited models and dependants run properly with production data: **N/A - there's no data :(**
- [X] The edited models are sufficiently documented.
- [ ] The edited models contain PK tests, and I've ran and passed them. **N/A - there's no data :(**
- [X] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models. **N/A - there's no data :(**
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19570
# Description
This PR fixes a bug in the deployment script. The bug makes the alerts not work as expected, because the bash piping will trigger alerts if the log writing of commands fail, instead of if commands fail, which is what is expected.
This PR modifies the way logging works. Logging remains the same, but alerts now will work as expected.
Related work items: #18183
# Description
This PR modifies the `docker compose` file that we recommend for local development, adding to it Postgres server parameters. This parameters are expected to provide a better experience for developers in terms of performance.
# Checklist
- NA ~~[ ] The edited models and dependants run properly with production data.~~
- NA ~~[ ] The edited models are sufficiently documented.~~
- NA ~~[ ] The edited models contain PK tests, and I've ran and passed them.~~
- NA ~~[ ] I have checked for DRY opportunities with other models and docs.~~
- NA ~~[ ] I've picked the right materialization for the affected models.~~
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #20193
# Description
Working version of User Product Bundle in intermediate. I tried to be quite explicit in the documentation of the model and the choices made (both in the code itself and in the schema). There's some opinionated choices so feel free to challenge them.
There's a small change on the user_migration model, in which I didn't properly set a field into a date.
Note that there's some schema comments pending from Lou's validation. Up to you if we prefer to wait until resolved or we move forward - to me, it's not blocking.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19570
# Description
Adds a missing field, `version`.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [ ] The edited models are sufficiently documented. -> __NO, but I promise they will be eventually__
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #20123
# Description
Creates exposure entries for the accounting reports that are already in production (Invoicing and Crediting and Resolutions Host Payments).
# Checklist
- __NA__ ~~[ ] The edited models and dependants run properly with production data.~~
- [X] The edited models are sufficiently documented.
- __NA__ ~~[ ] The edited models contain PK tests, and I've ran and passed them.~~
- __NA__ ~~[ ] I have checked for DRY opportunities with other models and docs.~~
- __NA__ ~~[ ] I've picked the right materialization for the affected models.~~
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #17550
# Description
First step towards reporting New Dash is to be able within DWH to know which hosts have been migrated.
In order to do so, and anticipating that there's going to be new phases in the future, I've created a `int_core__user_migration` model that reads from a configuration macro `get_new_dash_migration_phases_config` that will allow semi-automatic user retrieval in the future. This avoids nasty hardcoding within the model itself.
The information of whether a user is migrated, in which phase and when the phase was deployed is available at user level in the `int_core__user_host` table.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models. **-> I selected a view for this model since I don't believe we should materialse this data other than the user host table**
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19570
# Description
Sets the parameter to display the KPIs by number of listings to prod. I will move forward without the review as we need a simultaneous deployment. The combination of changes were reviewed yesterday in local.
# Checklist
- [ ] The edited models and dependants run properly with production data.
- [ ] The edited models are sufficiently documented.
- [ ] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19325
# Description
This PR turns the JSON documents of the verification container in sync into a table in staging. It also performs the necessary deduplication of records by ID and timestamp along the way.
I'm intentionally skipping proper documentation to unblock a colleague while I fetch the necessary knowledge to populate the schema docs properly. I pinky promise I won't forget and I'll come back and fix it.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [ ] ~~The edited models are sufficiently documented.~~ __Nope, read above.__
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #20123
# Description
Changes:
* Separate 1) the internal naming of dimensions available within DWH vs. 2) the display of the dimensions in the reporting. Mainly it changes the "by_number_of_listings" to display "By # of Listings Booked in 12 Months". I edited the production macro since to me it's linked to when things are available for display.
* Add preceding zeros on the segmentation so it's ordered correctly. Before, the segment 21-60 was displayed before the 6-20.
* Also added some capital letters to the schema config of the reporting model :)
I attach a screenshot of how it looks in PBI in my local development branch to exemplify why this is "Beautification". Be aware that merging this also puts in production the dimensions.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19325
# Description
This PR ensures the propagation of the dimensions for KPIs across the key aggregating and exposing models. Additionally, provides these 2 new fields in reporting while **not affecting the current data display**, thus it's safe to work in the PBI report without needing to work in 2 PRs in parallel.
**Changes:**
**1 - Intermediate, `int_mtd_vs_previous_year_metrics`:**
* Removes the temporary filter on `where dimension in ({{ production_dimensions }})`. This will be applied directly to reporting later. This ensures that the new dimension on customer segmentation is fully available only within intermediate.
* Adds `dimension` and `dimension_value` granularity. This includes: 1) adding these fields, 2) joining by these fields with all the source CTEs containing the source models with metrics - which in turn needs the change of the dates model - and 3) joining by these fields in the self-join to compute the incremental vs. previous year.
* Changes on the schema file
**2 - Intermediate, `int_mtd_aggregated_metrics`:**
* Adds `dimension` and `dimension_value` granularity. This includes only adding these fields.
* Changes on the schema file
**3 - Reporting, `mtd_aggregated_metrics`:**
* Adds the filter removed on `int_mtd_vs_previous_year_metrics`. This ensures that only the Global dimension is available for the reporting, thus **no changes from user POV**.
* Adds `dimension` and `dimension_value` granularity. This includes only adding these fields
* Changes on the schema file
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19325
# Description
Finished adding address_validation to reporting for check-in hero
# Checklist
- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [x] I have checked for DRY opportunities with other models and docs.
- [x] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
# Description
Added address_validation to int_core__vr_check_in_cover for check in hero report in PBI
# Checklist
- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [x] I have checked for DRY opportunities with other models and docs.
- [x] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #17069
# Description
Silly change:
* Modifies `int_core__mtd_guest_payments_metrics` to apply the proper key on date, dimension and dimension_value.
* -> *The weird thing is that the previous dbt test I run worked well. Is it possible that the configuration in the schema file prevails on top of the model configuration? I thought it was the other way around...*
Main changes:
* Modifies `int_xero__mtd_invoicing_metrics` to include the customer segmentation based on listings.
* `schema.yaml` is also affected including new fields and tests
* Added the macro to retrieve the production dimension in `int_core__mtd_vs_previous_year_metrics` to avoid propagating this upwards and messing up with the data display.
Overall, follows a similar strategy as we did for Booking, Guest Journey, Deal, Accommodation and Guest Payments metrics. For reference, [here's the previous PR on Guest Payments](https://guardhog.visualstudio.com/Data/_git/data-dwh-dbt-project/pullrequest/2580).
This is the last PR on the source models for KPIs. Will follow: propagation + exposure
# Checklist
- [X] The edited models and dependants run properly with production data.
* **Important note**: this segmentation provides null values for all API-related KPIs. Makes sense, since the 4 deal id we have for APIs do NOT have, or have had, a listing linked to them. I'd say it's not a blocker.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19325
# Description
Adds the possibility of considering as Hosts those users that come from Know Your Guest (KYG), after the discussion with Ben R yesterday. This uses the Claim table, specifically on any Kyg claim type:
- KygRegistrationSignUpType
- KygRegistrationIntegrationTypeName
- KygMvp
From what I see compiling the new vs. the previous version of this model and running into production to have up-to-date data, this increases the number of hosts in 8, from 2.608 to 2.616 so it's not a massive change in volumes.
I also modified the schema for this model to reflect the new logic.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19513
# Description
Created address_validations in intermediate for check in hero
# Checklist
- [X] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [x] I have checked for DRY opportunities with other models and docs.
- [x] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #17069
# Description
Modifies `int_core__mtd_guest_payments_metrics` to include the customer segmentation based on listings. `schema.yaml` is also affected including new fields, tests and apply the proper naming (from guest revenue to guest payments). I also modified a silly naming that was referring to deals to refer to listings/accommodations, my bad.
Added the macro to retrieve the production dimension in `int_core__mtd_vs_previous_year_metrics` to avoid propagating this upwards and messing up with the data display.
Overall, follows a similar strategy as we did for Booking, Guest Journey, Deal and Accommodation metrics. For reference, [here's the previous PR on Accommodations](https://guardhog.visualstudio.com/Data/_git/data-dwh-dbt-project/pullrequest/2575?_a=overview).
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19325
# Description
Takes into account @<Pablo Martín> 's feedback from the previous PR, slightly modified. This PR separates 1) the dimensions while developing vs. 2) the dimensions once these are available for production. This are within the same file of macro configuration for KPIs, namely `business_kpis_configuration`.
End-goal, all CTEs in `int_mtd_vs_previous_year_metrics` will read from this new macro `get_kpi_dimensions_for_production`, so eventually we won't need any hardcode once we want to add new dimensions. In the meantime, I'll be adding this new line for each PR (still 2 missing :D)
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [ ] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19325
# Description
Modifies `int_core__mtd_accommodation_metrics` to include the customer segmentation based on listings. `schema.yaml` is also affected including new fields and tests. Hardcoded `int_core__mtd_vs_previous_year_metrics` to avoid propagating this upwards and messing up with the data display.
Overall, follows a similar strategy as we did for Booking, Guest Journey and Deal metrics. For reference, here's [the previous PR on Deal](https://guardhog.visualstudio.com/Data/_git/data-dwh-dbt-project/pullrequest/2534). I noticed that I mixed the schema tests of Deals and Accommodations, this PR should fix both.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19325
# Description
New model for verification_to_address_validation for check in hero reporting
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #17069
