# Description
This relates to the data alerts on resolutions models.
We have one incident status that refers to the record having missing information. The record that is currently triggering this alerts is indeed tagged as such.
The idea is to:
* Create a new boolean that flags if the incident has missing information. This is propagated stg to reporting.
* Apply a where clause to any existing data alert that might be susceptible to fail due to not having enough quality, in staging and reporting.
* Remove over coverage in intermediate since it's already happening in reporting. However we keep a few not-null tests and the PK on critical information (i.e., id_booking).
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [NA] I have checked for DRY opportunities with other models and docs.
- [NA] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #31668
# Description
This relates to today's data alerts on Growth Score models.
Error has been found in `int_billable_items_growth_score_by_deal`. This only happens on the 1st of the month.
On the 1st day of the month we will have:
* The actual monthly data from KPIs in the previous month.
But also:
* Projected KPIs will still display data for the previous month, in this case though, projected = actuals.
There's a union all that combines actual monthly KPI data for history purposes and the projected data which fails in this case.
Fix is to remove the data from Projected KPIs if the current date of execution (1st of July) equals the first day of the month when computing `end_date` + 1 day. End date being at this moment 30th of June, + 1 day = 1st of July, extracting the first day of the month then 1st July.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [NA] I have checked for DRY opportunities with other models and docs.
- [NA] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #31665
# Description
Removed not_null tests for not mandatory fields, guest, host and listing data.
Manual forms are being excluded from the models because of the unreliability of the data
# Checklist
- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Removed not_null tests
Related work items: #31122
# Description
Fixes:
* There was a data quality issue in which active listings with bundles were showing more listings than the total active listings. This has been fixed, plus a bunch of tests have been added to make sure this does not happen again.
* There's a field that reaches the 63 character limit. I modified "paid" per "upgraded" which reduces this to intermediate model; however in reporting I need to keep as is to avoid breaking PBI. I did change the name of the field in the schema so tests/docs work well and explicitly mention this in the doc. This will need adaptation but can be done in the future.
* There was tons of dead code tagged as to be deleted that was not deleted, not sure why or when. Anyway this is dropped now. It's not used in PBI except for the fact that columns are dropped in the data model, so we might have a bit of down time to just remove these.
* Few fields were not displayed correctly in dbt docs.
What's new:
* New user categorisation as an alert-based system. This can be enhanced in the future but so far it covers hell of a lot of cases. This also includes data quality alerts (deal not in HS, or account defaults to MVP launch).
* MVP Launch date moved to project yaml since it's a very standard date.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #30407
# Description
We're getting Data Alerts triggered for Guest Journey payments with no id for the guest. This is because we have payments in the DWH for which we still don't have the related Verification Request in the DWH as well.
This PR changes a `LEFT JOIN` with the verification request table into an `INNER JOIN`, effectively making it impossible for a payment to not have user data if the VR is not there.
We increase data latency (the payment won't show up until the related VR is in the DWH) but we win consistency (no orphan payments without their full details).
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
change left join to inner join
# Description
New fields for Invoicing & Crediting that Finance needs for their usage.
Specifically they need to be able to differentiate the amount due because of invoice and not credit notes to know who to pay resolutions claims or not.
Will also add the amounts in local currency.
# Checklist
- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
New fields for Invoicing & Crediting
Related work items: #30204
# Description
This PR creates a proper Booking Source in `int_core__bookings` table. This mostly refers to ensure that whenever we do data requests tickets on this regards we have a proper logic in place (as last time we had issues because of this). Logic follows Gus recommendation on applying:
```
case
when upper(coalesce(bsrc.booking_source, 'UNKNOWN')) = 'UNKNOWN'
then b.raw_booking_source
else bsrc.booking_source
end
```
where b. is Booking table and bsrc. is Booking Source table.
Things to note:
* A new `stg_core__booking_source` table is created, full-refreshed from the backend (less than 10 records)
* I added a unique test for `stg_core__booking` that was not in there. I don't document the rest of the model though.
* I remove the previous `id_booking_source` as it wasn't used and the logic in place should be far more accurate.
* I do NOT remove the existing `verification_request_booking_source`. I don't remember what's this but I see this is in use specially on Guest Insights (CSAT) and the legacy Truvi reporting (Bookings, PMS, etc.). Not the scope of this ticket to change it.
* I don't do further propagation, but potentially, this could be extended to New Dash Booking Summary as this source is being shown in the dashboard itself.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #30415
# Description
Change in model `int_core__accommodation_to_product_bundle` to remove effective dates and use original start and end date.
Updated downstream models as well
# Checklist
- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
data alert fix
Related work items: #30948
# Description
This PR directly removes test accounts, and not just flag them, from:
* `int_core__unified_user`, which means is also propagated to `int_core__user_host`.
It does so by creating a new, extremely simple model named: `int_core__user_test_account`
Then, `int_core__user_host` is inner joined with a small explanation comment on:
* `int_core__accommodation`
* `int_core__unique_accommodation_to_user`
* `int_core__bookings`
* `int_core__verification_requests`
* `int_core__payments`
Effectively removing test accounts.
Lastly, any existence and usage of the column `is_test_account` has been removed -> mostly on New Dash-related models.
# Checklist
**As discussed in the daily, I'll be playing around in prod with this PR**
- [ ] The edited models and dependants run properly with production data.
- [ ] The edited models are sufficiently documented.
- [ ] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #27319
# Description
Applies the new dedicated source of truth of "is this account fake" based on Claims. This removes the previous logic.
In terms of differences, not much because:
1. New logic applied in the backend is pretty much the same as we had with a few improvements
2. This exclusion is only applied for New Dash. I checked the number of test accounts in New Dash and these are 2, the same ones in the new logic and for the old (currently in production). So no real impact.
Impact will come at the moment to start handling exclusions for KPIs, but this will done later on dedicated PRs to assess impacts properly.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #27319
# Description
This PR adds a new version to `int_core__check_in_cover_users`. The new version respectes the public interface of V1, but changes the strategy to determine which users get included in the list. Instead of relying on their account configuration (v1), v2 simply looks at whether they had any VR created offering CIH in the past 30 days.
This approach changes the semantic of the model, but we deem the new one reasonable.
The motivation behind this change is to ensure this models (1) keeps working after the activation of the Guest Products FF and (2) stays simple. Trying to follow a strategy where we look at both old and new configs was deemed very complex and didn't add much value.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #30165
# Description
Adds a field that helps categorize whether we owe money to an account or they owe money to Truvi.
Outstanding Resolutions or Outstanding Invoices
The idea is to include this field as a filter in the Invoicing & Crediting report to help Finance and Resolutions in the payment and approval process.
# Checklist
- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Outstanding Payments
Related work items: #30204
# Description
This PR moves the categorisation of Bookings with Incidents from the flagging specific model to Booking Summary, effectively keeping this at Booking ID level.
This also handles the direct dependency with the flagging categorisation model.
I also improved the documentation and test coverage on the Booking Summary as it's becoming more and more central to many areas.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #30676
# Description
This PR prepares the existing `int_core__vr_check_in_cover` model for the activation of the Guest Products feature flag.
It creates a second logic path which identifies VRs that have CIH in them ~~through product bundles~~ through looking at `stg_core__verification_request_to_guest_product `, and unions it with the old stuff.
The new CTE only gets called after the FF gets activated. This is defined in a var that we will need to adjust depending on when do we really activate the FF.
Note that these items are still pending:
- update the var to the right value once Lawrence pulls the trigger.
- clarify what the hell happens with CIH being "pushed" or not to ALL old dash hosts. This PR currently assumes this will NOT happen. If it will happen, we will need to change things.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #30165
