counterweight/data-dwh-dbt-project
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merged PR 5589: Limits data test coverage on resolutions data

Browse source 
# Description

This relates to the data alerts on resolutions models.

We have one incident status that refers to the record having missing information. The record that is currently triggering this alerts is indeed tagged as such.

The idea is to:
* Create a new boolean that flags if the incident has missing information. This is propagated stg to reporting.
* Apply a where clause to any existing data alert that might be susceptible to fail due to not having enough quality, in staging and reporting.
* Remove over coverage in intermediate since it's already happening in reporting. However we keep a few not-null tests and the PK on critical information (i.e., id_booking).

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [NA] I have checked for DRY opportunities with other models and docs.
- [NA] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #31668
This commit is contained in:
Oriol Roqué Paniagua 2025-07-01 09:44:54 +00:00
parent ca939e0c69
commit 8bc525e4c2
6 changed files with 99 additions and 141 deletions
Download patch file Download diff file Expand all files Collapse all files

1
models/intermediate/resolutions/int_resolutions__incidents.sql
View file

@ -147,6 +147,7 @@ select
i.id_incident,
i.id_verification,
i.current_status_name,
i.is_incident_missing_details,
i.is_submission_complete,
i.current_agent_name,
i.document_version,

125
models/intermediate/resolutions/schema.yml
View file

@ -26,6 +26,12 @@ models:
data_tests:
- not_null
- name: is_incident_missing_details
data_type: boolean
description: |
Flag to indicate if the incident is currently missing details.
This is used to limit the data tests coverage.
- name: is_submission_complete
data_type: boolean
description: "Flag to indicate if the submission is complete."
@ -74,10 +80,6 @@ models:
- name: time_to_first_comment_in_hours
data_type: numeric
description: "Time taken to reach the client for the first time."
data_tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: true
- name: first_status_at_utc
data_type: timestamp
@ -90,10 +92,6 @@ models:
- name: resolution_time_in_hours
data_type: numeric
description: "Time taken to resolve the incident, in hours, if completed."
data_tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: true
- name: number_damaged_items
data_type: bigint
@ -103,64 +101,35 @@ models:
data_type: numeric
description: |
"Original value of the damaged items in the host currency."
data_tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: original_value_amount_in_gbp
data_type: numeric
description: |
"Original value of the damaged items in GBP."
data_tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: asked_repair_amount_in_host_currency
data_type: numeric
description: |
"Amount requested by the host for repair in the host currency."
data_tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: asked_repair_amount_in_gbp
data_type: numeric
description: |
"Amount requested by the host for repair in GBP."
data_tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: accepted_amount_in_host_currency
data_type: numeric
description: |
"Final accepted settlement amount in the host currency."
data_tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: accepted_amount_in_gbp
data_type: numeric
description: |
"Final accepted settlement amount in GBP."
data_tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: host_currency
data_type: text
description: "Host or local currency"
data_tests:
- not_null:
where: "original_value_amount_in_host_currency > 0
or asked_repair_amount_in_host_currency > 0
or accepted_amount_in_host_currency > 0"
- name: guest_verification_status
data_type: text
@ -169,119 +138,63 @@ models:
- name: lower_protection_limit_usd
data_type: numeric
description: "Lower protection limit in USD."
data_tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: upper_protection_limit_usd
data_type: numeric
description: "Upper protection limit in USD."
data_tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: calculated_payout_amount_in_txn_currency
data_type: numeric
description: "Calculated payout amount in local currency."
data_tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: calculated_payout_currency
data_type: text
description: "Currency of the calculated payout amount."
data_tests:
- not_null:
where: "calculated_payout_amount_in_txn_currency > 0"
- name: calculated_payout_amount_in_gbp
data_type: numeric
description: "Calculated payout amount in GBP."
data_tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: submitted_payout_amount_in_txn_currency
data_type: numeric
description: "Submitted payout amount in local currency."
data_tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: submitted_payout_currency
data_type: text
description: "Currency of the submitted payout amount."
data_tests:
- not_null:
where: "submitted_payout_amount_in_txn_currency > 0"
- name: submitted_payout_amount_in_gbp
data_type: numeric
description: "Submitted payout amount in GBP."
data_tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: calculated_guest_charge_amount_in_txn_currency
data_type: numeric
description: "Calculated guest charge amount in local currency."
data_tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: calculated_guest_charge_currency
data_type: text
description: "Currency of the calculated guest charge amount."
data_tests:
- not_null:
where: "calculated_guest_charge_amount_in_txn_currency > 0"
- name: calculated_guest_charge_amount_in_gbp
data_type: numeric
description: "Calculated guest charge amount in GBP."
data_tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: submitted_guest_charge_amount_in_txn_currency
data_type: numeric
description: "Submitted guest charge amount in local currency."
data_tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: submitted_guest_charge_currency
data_type: text
description: "Currency of the submitted guest charge amount."
data_tests:
- not_null:
where: "submitted_guest_charge_amount_in_txn_currency > 0"
- name: submitted_guest_charge_amount_in_gbp
data_type: numeric
description: "Submitted guest charge amount in GBP."
data_tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: id_user_host
data_type: text
description: |
"Unique Id that identifies the host."
data_tests:
- relationships:
to: ref('stg_core__user')
field: id_user
- name: host_account_name
data_type: text
@ -310,10 +223,6 @@ models:
- name: id_user_guest
data_type: text
description: "Superhog code that uniquely identifies a single guest."
data_tests:
- relationships:
to: ref('stg_core__user')
field: id_user
- name: guest_email
data_type: text
@ -347,17 +256,10 @@ models:
- name: deposit_retained_amount_in_txn_currency
data_type: numeric
description: "Amount of the deposit retained in local currency."
data_tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: deposit_retained_currency
data_type: text
description: "Currency of the deposit retained."
data_tests:
- not_null:
where: "deposit_retained_amount_in_txn_currency > 0"
- name: has_guest_contributed_to_cost
data_type: boolean
@ -373,26 +275,15 @@ models:
data_type: numeric
description: "Amount of the guest contribution, in case they did,
in local currency."
data_tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: guest_contribution_currency
data_type: text
description: "Currency of the guest contribution."
data_tests:
- not_null:
where: "guest_contribution_amount_in_txn_currency > 0"
- name: guest_contribution_amount_in_gbp
data_type: numeric
description: "Amount of the guest contribution, in case they did,
in GBP."
data_tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: is_guest_contacted_about_damage
data_type: boolean
@ -401,10 +292,6 @@ models:
- name: id_accommodation
data_type: numeric
description: "Superhog code that uniquely identifies a single accommodation."
data_tests:
- relationships:
to: ref('stg_core__accommodation')
field: id_accommodation
- name: accommodation_name
data_type: text