# Description
This PR fixes today's issue on dbt test, since we had a duplicated booking in this table.
Technically, according to Daga, a booking can only be overridden with a new program ONLY if the first program was the default BasicScreening, and it's a new bundle containing a Paid Service. Once there's a product bundle applied to a booking that contains a paid service, it cannot be overridden.
Thus in essence, by just keeping the latest update date it will only take into account the potential switch to a paid service, that is what really matters, and cannot change afterwards.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #22346
# Description
Creation on stg_hubspot__form_submissions model
# Checklist
- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Stg Hubspot form_submissions
# Description
Creation on stg_hubspot__contacts model
# Checklist
- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
staging Hubspot contacts
# Description
Quick fix to make lower_limit_migration_date_utc more robust for new dash reporting purposes. I noticed some cases in which users were actually created after the migration date. In these cases, I just keep joined date.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #22331
# Description
This PR creates 3 new models in staging:
- `stg_core__protection_plan`: main information of a protection plan. Here I denormalise the info coming from Protection, meaning, Protection is not a standalone model.
- `stg_core__protection_plan_to_price`: How much does a protection service cost? It's very similar to the model product_service_to_price.
- `stg_core__protection_plan_cover`: How much does a protection service protects, i.e., covers? I took the liberty of modifying the original name since it was confusing to me. Let me know if you'd like to modify the name.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #20809
# Description
This PR adapts the outlier detection test for KPIs. Specifically:
1) It removes not additive lifecycle metrics, which are:
- Churning Listings/Deals
- Listings/Deals Booked in Month/6 Months/12 Months
this is because the test computes data at daily level by just doing value/number of days. The thing is that for all these metrics, Listing/Deal bookings are computed **uniquely over a month**, i.e., if a listing is booked 100 times in a single month, it will only appear as once. Thus it makes it fail on early days of the month. Similar case for Churn, in this case, at the beginning of the month we have the total maximum number of listing/deals that are expected to churn if nothing happens, and this can decrease a bit over time if these get reactivated.
2) I reduced the variance threshold from 10 to 8, meaning now the alerts will raise more often. This is because we're removing some wrongly assessed metrics from the computation, thus I feel we can leave with better fine-grained detection. It could be even further reduced (8 is still super high tolerance) since today maximum signal-to-noise ratio was less than 4 on checkout bookings, but I'd propose to see how it goes in the following days and then assess if it's necessary to reduce it even further.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [ ] The edited models are sufficiently documented.
- [ ] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
# Description
This PR adds a new table named `user_product_bundle_contains_services` in intermediate core.
Mainly, this table serves as a bridge between `user_product_bundle` and `product_services`. A `product_bundle` within `user_product_bundle` can contain 1 or several services, and this was stated in the field `contains_product_services`. The value of this field corresponds to the sum of `product_service_binary_tier` from the services that apply within that bundle. Even though the information is quite concise using this format, it adds extra complexity for analytical purposes, so this new table just duplicates the `user_product_bundle` main attributes as many times as services are contained.
For example:
`id_user_product_bundle` = 383 contains one unique bundle named Basic Program. This bundle has the `chosen_product_services` = 257, which can only be decomposed in power of 2 as of the sum of 256 + 1. This bundle therefore contains 2 services, Basic Screening (`product_service_binary_tier` = 1) and Waiver Pro (`product_service_binary_tier` = 256). Thus, in the new table, we will have 2 records and remove all this logic to something more standard, as seen in this screenshot:
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [ ] The edited models contain PK tests, and I've ran and passed them. *N/A - there's no PK in this table, but the combination of columns that should be unique is tested*
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #20809
# Description
This PR integrates into staging the following Core tables, regarding New Pricing:
- ProductService
- ProductServiceToPrice
- AppliedProductService (IT HAS BEEN REMOVED)
Models are also documented in staging. Tables contain data already but not necessarily records for New Dash, since still we have not had any paid service applied into a booking / V2 has not been launched yet.
New Dash modelisation for V2 will follow once these tables exist in staging, in order to be able to report 1) revenue and 2) services selected.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models. **N/A, few records so not a big deal. Might need to change in the future.**
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #20809
# Description
Activates a commented out test for verification payments v2. The test was deactivated because of some master data issues in the backend, but they were fixed after #22148.
# Checklist
- [NA] The edited models and dependants run properly with production data.
- [NA ] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [NA] I have checked for DRY opportunities with other models and docs.
- [NA] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #20043
# Description
Included more information in the reporting model to add more details in the report
# Checklist
- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [x] I have checked for DRY opportunities with other models and docs.
- [x] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
more features into reporting
# Description
Added 2 new columns with a concatenation of month number and full name o short name
# Checklist
- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [ ] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
new features dates
# Description
Fixed cancellation fee charge that was being applied to every verification when the threshold was passed, now it will be applied only to those that are actually cancelled
# Checklist
- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
fixed cancellation fee charge
# Description
This PR adds Kosovo to our guest vat rates seed data. It's an exceptional record because it doesn't comply with the ISO 3166 that the model is supposed to follow. But we need to do it because Kosovo is a recognized country in the Superhog backend.
Nila from finance confirmed that the VAT rate should be 0.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #22148
# Description
Creation on stg_hubspot__deals model
# Checklist
- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
# Description
There were two tests for `int_core__verification_payments` awaiting activation because the source master data had issues that needed to be fixed.
This PR activates the test for `is_missing_user_country` because the dash team has already fixed the problematic country data.
The other test remains turned off because we still have a master data issue (missing VAT rate for Kosovo), but I've already added a missing where condition, and a reference to the ticket where we are tracking the master data issue.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
activate one test, prepare another but keep it commented
Related work items: #20043
# Description
Added new role to int_core__user_role
# Checklist
- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Added new role
# Description
Fixes a bug introduced by PR: https://guardhog.visualstudio.com/Data/_git/data-pbi-reports/pullrequest/2962
The previous PR made non-waiver services become tax free if the host had an active payaway setting at the payment timestamp, which makes no sense because payaway only relates to waiver and not to other services.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #20043
# Description
This PR integrates screening API verifications into staging.
There's 2 commits:
* The earliest one, it just copy-pastes the strategy followed by edeposit and adapts it to fit screening API case, which is simpler. Note here that the schema entries contain a low number of tests. This is because we only have 7 records in production - and these seem fake anyway :) - so it's complex to extrapolate. Those I could extrapolate (NoFlags/Flagged) I'm taking from the data transformation within the PBI report.
* The last commit, it's just a DRY. It handles the deduplication logic for cosmos db in a macro, and I applied it on both the screening API and the edeposit verifications. Works well from my POV.
Since you guys are far more knowledgeable on APIs scope, I'll let you take a close look in case I missed something.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [ ] The edited models are sufficiently documented. **I guess it can be improved, waiting for your comments here**
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models. **Used default coming from stg_edeposit__verifications**
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #20127
# Description
This PR modifies the V2 of `int_core__verification_payments` to make sure that taxes are not computed for waivers where the host takes the risk (in line with our guest taxes guidelines: https://www.notion.so/knowyourguest-superhog/Guest-Services-Taxes-How-to-calculate-a5ab4c049d61427fafab669dbbffb3a2?pvs=4#323e230365ed496cad6a65b5659895ea).
The PR changes the intermediate column `is_service_subject_to_vat`, making it `false` in the cases where there was no active payaway plan during the month where the payment was generated. This cascades in the main body of the model, because if `is_service_subject_to_vat` is `false`, then the tax amounts get turned into zero.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
add logic
Related work items: #20043
# Description
Changed the financial rate metrics to have `number_fomat = "converted_metric_currency_gbp"`
# Checklist
- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Financial rate metrics
Related work items: #21564
