counterweight/data-dwh-dbt-project
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merged PR 5589: Limits data test coverage on resolutions data

Browse source 
# Description

This relates to the data alerts on resolutions models.

We have one incident status that refers to the record having missing information. The record that is currently triggering this alerts is indeed tagged as such.

The idea is to:
* Create a new boolean that flags if the incident has missing information. This is propagated stg to reporting.
* Apply a where clause to any existing data alert that might be susceptible to fail due to not having enough quality, in staging and reporting.
* Remove over coverage in intermediate since it's already happening in reporting. However we keep a few not-null tests and the PK on critical information (i.e., id_booking).

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [NA] I have checked for DRY opportunities with other models and docs.
- [NA] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #31668
This commit is contained in:
Oriol Roqué Paniagua 2025-07-01 09:44:54 +00:00
parent ca939e0c69
commit 8bc525e4c2
6 changed files with 99 additions and 141 deletions
Download patch file Download diff file Expand all files Collapse all files

51
models/reporting/resolutions/schema.yml
View file

@ -26,6 +26,14 @@ models:
data_tests:
- not_null
- name: is_incident_missing_details
data_type: boolean
description: |
Flag to indicate if the incident is currently missing details.
This is used to limit the data tests coverage.
data_tests:
- not_null
- name: is_submission_complete
data_type: boolean
description: "Flag to indicate if the submission is complete."
@ -78,6 +86,7 @@ models:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: true
where: not is_incident_missing_details
- name: first_status_at_utc
data_type: timestamp
@ -94,6 +103,7 @@ models:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: true
where: not is_incident_missing_details
- name: number_damaged_items
data_type: bigint
@ -107,6 +117,7 @@ models:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
where: not is_incident_missing_details
- name: original_value_amount_in_gbp
data_type: numeric
@ -116,6 +127,7 @@ models:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
where: not is_incident_missing_details
- name: asked_repair_amount_in_host_currency
data_type: numeric
@ -125,6 +137,7 @@ models:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
where: not is_incident_missing_details
- name: asked_repair_amount_in_gbp
data_type: numeric
@ -134,6 +147,7 @@ models:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
where: not is_incident_missing_details
- name: accepted_amount_in_host_currency
data_type: numeric
@ -143,6 +157,7 @@ models:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
where: not is_incident_missing_details
- name: accepted_amount_in_gbp
data_type: numeric
@ -152,15 +167,17 @@ models:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
where: not is_incident_missing_details
- name: host_currency
data_type: text
description: "Host or local currency"
data_tests:
- not_null:
where: "original_value_amount_in_host_currency > 0
where: "(original_value_amount_in_host_currency > 0
or asked_repair_amount_in_host_currency > 0
or accepted_amount_in_host_currency > 0"
or accepted_amount_in_host_currency > 0)
and not is_incident_missing_details"
- name: guest_verification_status
data_type: text
@ -173,6 +190,7 @@ models:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
where: not is_incident_missing_details
- name: upper_protection_limit_usd
data_type: numeric
@ -181,6 +199,7 @@ models:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
where: not is_incident_missing_details
- name: calculated_payout_amount_in_txn_currency
data_type: numeric
@ -189,13 +208,14 @@ models:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
where: not is_incident_missing_details
- name: calculated_payout_currency
data_type: text
description: "Currency of the calculated payout amount."
data_tests:
- not_null:
where: "calculated_payout_amount_in_txn_currency > 0"
where: "calculated_payout_amount_in_txn_currency > 0 and not is_incident_missing_details"
- name: calculated_payout_amount_in_gbp
data_type: numeric
@ -204,6 +224,7 @@ models:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
where: not is_incident_missing_details
- name: submitted_payout_amount_in_txn_currency
data_type: numeric
@ -212,13 +233,14 @@ models:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
where: not is_incident_missing_details
- name: submitted_payout_currency
data_type: text
description: "Currency of the submitted payout amount."
data_tests:
- not_null:
where: "submitted_payout_amount_in_txn_currency > 0"
where: "submitted_payout_amount_in_txn_currency > 0 and not is_incident_missing_details"
- name: submitted_payout_amount_in_gbp
data_type: numeric
@ -227,6 +249,7 @@ models:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
where: not is_incident_missing_details
- name: calculated_guest_charge_amount_in_txn_currency
data_type: numeric
@ -235,13 +258,14 @@ models:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
where: not is_incident_missing_details
- name: calculated_guest_charge_currency
data_type: text
description: "Currency of the calculated guest charge amount."
data_tests:
- not_null:
where: "calculated_guest_charge_amount_in_txn_currency > 0"
where: "calculated_guest_charge_amount_in_txn_currency > 0 and not is_incident_missing_details"
- name: calculated_guest_charge_amount_in_gbp
data_type: numeric
@ -250,6 +274,7 @@ models:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
where: not is_incident_missing_details
- name: submitted_guest_charge_amount_in_txn_currency
data_type: numeric
@ -258,13 +283,14 @@ models:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
where: not is_incident_missing_details
- name: submitted_guest_charge_currency
data_type: text
description: "Currency of the submitted guest charge amount."
data_tests:
- not_null:
where: "submitted_guest_charge_amount_in_txn_currency > 0"
where: "submitted_guest_charge_amount_in_txn_currency > 0 and not is_incident_missing_details"
- name: submitted_guest_charge_amount_in_gbp
data_type: numeric
@ -273,6 +299,7 @@ models:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
where: not is_incident_missing_details
- name: id_user_host
data_type: text
@ -282,6 +309,7 @@ models:
- relationships:
to: ref('stg_core__user')
field: id_user
where: not is_incident_missing_details
- name: host_account_name
data_type: text
@ -314,6 +342,7 @@ models:
- relationships:
to: ref('stg_core__user')
field: id_user
where: not is_incident_missing_details
- name: guest_email
data_type: text
@ -351,13 +380,14 @@ models:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
where: not is_incident_missing_details
- name: deposit_retained_currency
data_type: text
description: "Currency of the deposit retained."
data_tests:
- not_null:
where: "deposit_retained_amount_in_txn_currency > 0"
where: "deposit_retained_amount_in_txn_currency > 0 and not is_incident_missing_details"
- name: has_guest_contributed_to_cost
data_type: boolean
@ -377,13 +407,14 @@ models:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
where: not is_incident_missing_details
- name: guest_contribution_currency
data_type: text
description: "Currency of the guest contribution."
data_tests:
- not_null:
where: "guest_contribution_amount_in_txn_currency > 0"
where: "guest_contribution_amount_in_txn_currency > 0 and not is_incident_missing_details"
- name: guest_contribution_amount_in_gbp
data_type: numeric
@ -393,6 +424,7 @@ models:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
where: not is_incident_missing_details
- name: is_guest_contacted_about_damage
data_type: boolean
@ -405,6 +437,7 @@ models:
- relationships:
to: ref('stg_core__accommodation')
field: id_accommodation
where: not is_incident_missing_details
- name: accommodation_name
data_type: text
@ -441,4 +474,4 @@ models:
- name: id_reservation
data_type: text
description: "External unique ID for the reservation."
description: "External unique ID for the reservation."