counterweight/data-dwh-dbt-project
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merged PR 5366: Adds Incident data to Booking Summary

Browse source 
# Description

This PR moves the categorisation of Bookings with Incidents from the flagging specific model to Booking Summary, effectively keeping this at Booking ID level.

This also handles the direct dependency with the flagging categorisation model.

I also improved the documentation and test coverage on the Booking Summary as it's becoming more and more central to many areas.

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #30676
This commit is contained in:
Oriol Roqué Paniagua 2025-06-02 15:56:54 +00:00
parent 92b144d78e
commit affeac612a
3 changed files with 453 additions and 258 deletions
Download patch file Download diff file Expand all files Collapse all files

133
models/intermediate/cross/int_booking_summary.sql
View file

@ -1,3 +1,21 @@
{% set risk_booking_status = ("NOTAPPROVED", "FLAGGED") %}
{% set no_risk_booking_status = ("APPROVED", "NOFLAGS") %}
{% set incident_duplicated_status = "CLOSED - DUPLICATE" %}
{% set incident_finished_status = (
"RESOLVED",
"RESOLVED EXCEPTION",
"CLOSED - NO REPLY",
"CLOSED - OTHER",
"CLOSED - WAIVER CR",
"CLOSED - LATE REPORT",
"CLOSED - NOT COVERED",
"CLOSED - NOT LIABLE",
"CLOSED HOST REQUEST",
"CLOSED - NOT APPROVED",
"CLOSED - THIRD PARTY",
) %}
{% set days_from_checkout_to_completion = 14 %}
{{ config(materialized="table", unique_key=["id_booking"]) }} {{ config(materialized="table", unique_key=["id_booking"]) }}
with with
int_core__booking_to_service as ( int_core__booking_to_service as (
@ -5,9 +23,14 @@ with
), ),
int_core__booking_service_detail as ( int_core__booking_service_detail as (
select * from {{ ref("int_core__booking_service_detail") }} select * from {{ ref("int_core__booking_service_detail") }}
) ),
int_resolutions__incidents as (
select select * from {{ ref("int_resolutions__incidents") }}
),
-- This CTE handles the core backend booking information and aggregates service
-- details.
core_backend_booking_modelling as (
select
bts.id_booking, bts.id_booking,
bts.id_verification_request, bts.id_verification_request,
bts.id_accommodation, bts.id_accommodation,
@ -25,14 +48,20 @@ select
bts.booking_check_in_date_utc, bts.booking_check_in_date_utc,
bts.booking_check_out_at_utc, bts.booking_check_out_at_utc,
bts.booking_check_out_date_utc, bts.booking_check_out_date_utc,
bts.booking_check_out_date_utc
+ {{ days_from_checkout_to_completion }} as booking_completed_date_utc,
bts.booking_number_of_nights, bts.booking_number_of_nights,
bts.host_currency_code, bts.host_currency_code,
bts.is_user_in_new_dash, bts.is_user_in_new_dash,
bts.new_dash_version, bts.new_dash_version,
bts.user_in_new_dash_since_timestamp_at_utc, bts.user_in_new_dash_since_timestamp_at_utc,
sum(bsd.service_total_price_in_gbp) as booking_total_price_in_gbp, sum(bsd.service_total_price_in_gbp) as booking_total_price_in_gbp,
min(bsd.service_first_chargeable_date_utc) as service_first_chargeable_date_utc, min(
max(bsd.service_last_chargeable_date_utc) as service_last_chargeable_date_utc, bsd.service_first_chargeable_date_utc
) as service_first_chargeable_date_utc,
max(
bsd.service_last_chargeable_date_utc
) as service_last_chargeable_date_utc,
min(bsd.service_first_billable_date_utc) as service_first_billable_date_utc, min(bsd.service_first_billable_date_utc) as service_first_billable_date_utc,
max(bsd.service_last_billable_date_utc) as service_last_billable_date_utc, max(bsd.service_last_billable_date_utc) as service_last_billable_date_utc,
min(bsd.service_detail_created_at_utc) as service_first_created_at_utc, min(bsd.service_detail_created_at_utc) as service_first_created_at_utc,
@ -41,17 +70,23 @@ select
count(distinct bsd.id_booking_service_detail) as number_of_applied_services, count(distinct bsd.id_booking_service_detail) as number_of_applied_services,
count( count(
distinct case distinct case
when bsd.is_paid_service then bsd.id_booking_service_detail else null when bsd.is_paid_service
then bsd.id_booking_service_detail
else null
end end
) as number_of_applied_paid_services, ) as number_of_applied_paid_services,
count( count(
distinct case distinct case
when bsd.is_upgraded_service then bsd.id_booking_service_detail else null when bsd.is_upgraded_service
then bsd.id_booking_service_detail
else null
end end
) as number_of_applied_upgraded_services, ) as number_of_applied_upgraded_services,
count( count(
distinct case distinct case
when bsd.is_billable_service then bsd.id_booking_service_detail else null when bsd.is_billable_service
then bsd.id_booking_service_detail
else null
end end
) as number_of_applied_billable_services, ) as number_of_applied_billable_services,
case case
@ -105,7 +140,12 @@ select
end as has_billable_services, end as has_billable_services,
case case
when when
sum(case when bsd.service_business_type = 'SCREENING' then 1 else 0 end) > 0 sum(
case
when bsd.service_business_type = 'SCREENING' then 1 else 0
end
)
> 0
then true then true
else false else false
end as has_screening_service_business_type, end as has_screening_service_business_type,
@ -128,7 +168,9 @@ select
when when
sum( sum(
case case
when bsd.service_business_type = 'DEPOSIT_MANAGEMENT' then 1 else 0 when bsd.service_business_type = 'DEPOSIT_MANAGEMENT'
then 1
else 0
end end
) )
> 0 > 0
@ -137,7 +179,11 @@ select
end as has_deposit_management_service_business_type, end as has_deposit_management_service_business_type,
case case
when when
sum(case when bsd.service_business_type = 'PROTECTION' then 1 else 0 end) sum(
case
when bsd.service_business_type = 'PROTECTION' then 1 else 0
end
)
> 0 > 0
then true then true
else false else false
@ -145,12 +191,26 @@ select
bts.is_missing_id_deal, bts.is_missing_id_deal,
case case
when bts.host_currency_code is null then true else false when bts.host_currency_code is null then true else false
end as is_missing_host_currency_code end as is_missing_host_currency_code,
from int_core__booking_to_service bts case
inner join when
(current_date - booking_check_out_date_utc)
> {{ days_from_checkout_to_completion }}
then true
else false
end as is_booking_past_completion_date,
case
when upper(bts.booking_status) in {{ risk_booking_status }}
then true
when upper(bts.booking_status) in {{ no_risk_booking_status }}
then false
else null
end as is_booking_flagged_as_risk
from int_core__booking_to_service bts
inner join
int_core__booking_service_detail bsd int_core__booking_service_detail bsd
on bts.id_booking_service_detail = bsd.id_booking_service_detail on bts.id_booking_service_detail = bsd.id_booking_service_detail
group by group by
bts.id_booking, bts.id_booking,
bts.id_verification_request, bts.id_verification_request,
bts.id_accommodation, bts.id_accommodation,
@ -174,3 +234,46 @@ group by
bts.is_user_in_new_dash, bts.is_user_in_new_dash,
bts.new_dash_version, bts.new_dash_version,
bts.user_in_new_dash_since_timestamp_at_utc bts.user_in_new_dash_since_timestamp_at_utc
),
-- This CTE deduplicates resolution incidents by booking ID and aggregates payout
-- amounts.
deduplicated_incidents as (
select
id_booking,
sum(submitted_payout_amount_in_gbp) as submitted_payout_amount_in_gbp,
case
when sum(submitted_payout_amount_in_gbp) > 0 then true else false
end as has_submitted_payout,
case
when
sum(
case
when
upper(current_status_name)
in {{ incident_finished_status }}
then 1
else 0
end
)
> 0
then true
else false
end as is_incident_finished
from int_resolutions__incidents
where upper(current_status_name) != '{{ incident_duplicated_status }}'
group by 1
)
-- This CTE combines the core backend booking information with the deduplicated
-- incidents to provide a comprehensive booking summary.
select
-- Retrieve all previous backend booking modelling fields.
cbbm.*,
-- Retrieve incident resolution fields.
case
when di.id_booking is not null then true else false
end as has_resolution_incident,
coalesce(di.is_incident_finished, false) as has_resolution_incident_finished,
coalesce(di.has_submitted_payout, false) as has_resolution_submitted_payout,
submitted_payout_amount_in_gbp as resolution_submitted_payout_amount_in_gbp
from core_backend_booking_modelling cbbm
left join deduplicated_incidents di on cbbm.id_booking = di.id_booking

94
models/intermediate/cross/int_flagging_booking_categorisation.sql
View file

@ -1,71 +1,17 @@
{% set risk_booking_status = ("NOTAPPROVED", "FLAGGED") %}
{% set no_risk_booking_status = ("APPROVED", "NOFLAGS") %}
{% set incident_duplicated_status = "CLOSED - DUPLICATE" %}
{% set incident_finished_status = (
"RESOLVED",
"RESOLVED EXCEPTION",
"CLOSED - NO REPLY",
"CLOSED - OTHER",
"CLOSED - WAIVER CR",
"CLOSED - LATE REPORT",
"CLOSED - NOT COVERED",
"CLOSED - NOT LIABLE",
"CLOSED HOST REQUEST",
"CLOSED - NOT APPROVED",
"CLOSED - THIRD PARTY",
) %}
{% set days_from_checkout_to_completion = 14 %}
{{ config(materialized="table") }} {{ config(materialized="table") }}
with with
int_booking_summary as (select * from {{ ref("int_booking_summary") }}), int_booking_summary as (select * from {{ ref("int_booking_summary") }}),
int_resolutions__incidents as (
select * from {{ ref("int_resolutions__incidents") }}
),
-- The same booking can have multiple Incidents
deduplicated_incidents as (
select
id_booking,
sum(submitted_payout_amount_in_gbp) as submitted_payout_amount_in_gbp,
case
when sum(submitted_payout_amount_in_gbp) > 0 then true else false
end as has_submitted_payout,
case
when
sum(
case
when
upper(current_status_name)
in {{ incident_finished_status }}
then 1
else 0
end
)
> 0
then true
else false
end as is_incident_finished
from int_resolutions__incidents
where upper(current_status_name) != '{{ incident_duplicated_status }}'
group by 1
),
new_dash_protected_bookings as ( new_dash_protected_bookings as (
select select
id_booking, id_booking,
case is_booking_past_completion_date as is_booking_completed,
when is_booking_flagged_as_risk,
(current_date - booking_check_out_date_utc) has_resolution_incident as has_claim,
> {{ days_from_checkout_to_completion }} has_resolution_submitted_payout as has_submitted_payout,
then true has_resolution_incident_finished as is_incident_finished,
else false coalesce(
end as is_booking_completed, resolution_submitted_payout_amount_in_gbp, 0
case ) as submitted_payout_amount_in_gbp
when upper(booking_status) in {{ risk_booking_status }}
then true
when upper(booking_status) in {{ no_risk_booking_status }}
then false
else null
end as is_booking_flagged_as_risk
from int_booking_summary from int_booking_summary
where where
-- Bookings from New Dash users with Id Deal -- Bookings from New Dash users with Id Deal
@ -76,25 +22,9 @@ with
has_protection_service_business_type has_protection_service_business_type
or has_deposit_management_service_business_type or has_deposit_management_service_business_type
) )
-- Bookings with relevant status (i.e. not cancelled, not pending) -- Bookings with flagging categorisation (this excludes
and ( -- Cancelled/Incomplete Information/Rejected bookings)
upper(booking_status) in {{ risk_booking_status }} and is_booking_flagged_as_risk is not null
or upper(booking_status) in {{ no_risk_booking_status }}
)
),
bookings_with_incidents as (
select
ndpb.id_booking,
ndpb.is_booking_completed,
ndpb.is_booking_flagged_as_risk,
case when di.id_booking is not null then true else false end as has_claim,
coalesce(di.has_submitted_payout, false) as has_submitted_payout,
coalesce(di.is_incident_finished, false) as is_incident_finished,
coalesce(
submitted_payout_amount_in_gbp, 0
) as submitted_payout_amount_in_gbp
from new_dash_protected_bookings ndpb
left join deduplicated_incidents di on ndpb.id_booking = di.id_booking
) )
select select
-- High Level Bookings -- -- High Level Bookings --
@ -275,4 +205,4 @@ select
and is_incident_finished and is_incident_finished
) as completed_no_risk_with_submitted_payout_amount_paid_in_gbp ) as completed_no_risk_with_submitted_payout_amount_paid_in_gbp
from bookings_with_incidents from new_dash_protected_bookings

182
models/intermediate/cross/schema.yml
View file

@ -3406,20 +3406,42 @@ models:
- name: int_booking_summary - name: int_booking_summary
description: | description: |
This model contains enriched information aggregated at Booking level regarding This model contains enriched information aggregated at Booking level under New Pricing
the services that are applied within a Booking. regarding the services that are applied within a Booking.
Specifically, contains both Booking and Services attributes (aggregated), as well
as the total price in GBP at this specific moment in time. In other words,
it's the snapshot of the current status of the Booking.
It's a subset of all bookings since it only applies to bookings that come from
hosts that have been migrated into the New Dash or New Pricing.
If you want to retrieve only New Dash Bookings, you will need to apply the following Specifically, contains both Booking and Services attributes (aggregated), as well
filters: as the total price in GBP at this specific moment in time. This model also contains
information on the potential Resolution Incidents linked to a Booking, as long as
these appear in the Resolution Center.
In other words, it's the snapshot of the current status of the Booking.
Be aware that this model only contains a subset of all bookings. It only applies to
1) Platform (Dashboard) bookings, so APIs are excluded and,
2) that come from hosts that have been migrated into the New Pricing.
In essence, that the Booking is linked to a User Product Bundle.
It's likely that you are just interested in retrieveing only New Dash Bookings. For this,
you will need to apply the following filters:
- is_user_in_new_dash = True - is_user_in_new_dash = True
- is_missing_id_deal = False - is_missing_id_deal = False
For Data-Driven Flagging purposes, you're likely going to be interested in applying all or
part of the following filters:
1. Bookings from New Dash users with Id Deal
- is_user_in_new_dash = True
- is_missing_id_deal = False
2. Protected Bookings with a Protection or a Deposit Management service
- has_protection_service_business_type or has_deposit_management_service_business_type
3. Bookings with flagging categorisation (this excludes Cancelled/Incomplete/Rejected bookings)
- is_booking_flagged_as_risk is not null
data_tests: data_tests:
- dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B:
column_A: booking_completed_date_utc
column_B: booking_check_out_at_utc
or_equal: False
- dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B: - dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B:
column_A: booking_check_out_at_utc column_A: booking_check_out_at_utc
column_B: booking_check_in_at_utc column_B: booking_check_in_at_utc
@ -3490,6 +3512,15 @@ models:
The current status of the booking. Cannot be null. The current status of the booking. Cannot be null.
data_tests: data_tests:
- not_null - not_null
- accepted_values:
values:
- APPROVED
- NOFLAGS
- NOTAPPROVED
- FLAGGED
- CANCELLED
- INCOMPLETEINFORMATION
- REJECTED
- name: program_name - name: program_name
data_type: string data_type: string
@ -3562,6 +3593,16 @@ models:
data_tests: data_tests:
- not_null - not_null
- name: booking_completed_date_utc
data_type: date
description: |
Date of the Booking completion. This is assumed to be 14 days
after Check-out date, and it is used to determine the limit in
which the Host can raise a Resolution Incident regarding this
booking.
data_tests:
- not_null
- name: booking_number_of_nights - name: booking_number_of_nights
data_type: integer data_type: integer
description: | description: |
@ -3661,24 +3702,51 @@ models:
data_type: integer data_type: integer
description: | description: |
Total number of Services applied to this Booking. Total number of Services applied to this Booking.
data_tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: true
- name: number_of_applied_paid_services - name: number_of_applied_paid_services
data_type: integer data_type: integer
description: | description: |
Total number of Services that require a monetary Total number of Services that require a monetary
income to Superhog applied to this Booking. income to Superhog applied to this Booking.
data_tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
max_value: number_of_applied_upgraded_services
strictly: false
- name: number_of_applied_upgraded_services - name: number_of_applied_upgraded_services
data_type: integer data_type: integer
description: | description: |
Total number of Services different from Basic Screening Total number of Services different from Basic Screening
applied to this Booking. applied to this Booking.
data_tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
max_value: number_of_applied_services
strictly: false
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: true
where: (has_protection_service_business_type = True or has_deposit_management_service_business_type = True)
- name: number_of_applied_billable_services - name: number_of_applied_billable_services
data_type: integer data_type: integer
description: | description: |
Total number of Services that require an invoice to the Host Total number of Services that require an invoice to the Host
to the Host, that are applied to this Booking. that are applied to this Booking.
data_tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
max_value: number_of_applied_upgraded_services
strictly: false
- name: is_booking_chargeable - name: is_booking_chargeable
data_type: boolean data_type: boolean
@ -3784,3 +3852,97 @@ models:
description: | description: |
Flag to identify if the Host for this booking is missing Flag to identify if the Host for this booking is missing
the currency code or not. the currency code or not.
- name: is_booking_past_completion_date
data_type: boolean
description: |
Flag to identify if the booking is past the completion date or not.
This is used to determine if the booking can be considered as completed
or not, and therefore, if it can be used for further risk-assessment
analysis.
data_tests:
- not_null
- name: is_booking_flagged_as_risk
data_type: boolean
description: |
Flag to identify if the booking has been flagged as risk or not.
This is based on the Booking Status and will return:
- True if the status is "NOTAPPROVED" or "FLAGGED"
- False if the status is "APPROVED" or "NOFLAGS"
- Null otherwise, such as "CANCELLED", "INCOMPLETEINFORMATION" or "REJECTED"
data_tests:
- accepted_values:
values:
- true
where: (upper(booking_status) in ('NOTAPPROVED', 'FLAGGED'))
- accepted_values:
values:
- false
where: (upper(booking_status) in ('APPROVED', 'NOFLAGS'))
- dbt_expectations.expect_column_values_to_be_null:
where: (upper(booking_status) not in ('NOTAPPROVED', 'FLAGGED', 'APPROVED', 'NOFLAGS'))
- name: has_resolution_incident
data_type: boolean
description: |
Flag to identify if the booking has a Resolution Incident or not. This might
imply that the resolution is in progress or that has been resolved, so it's not
necessarily finished.
If True, then the booking has at least one Resolution Incident.
If False, then the booking has no Resolution Incidents - but this does not
guarantee that the booking might have a resolution incident in the future.
data_tests:
- not_null
- accepted_values:
values:
- true
where: (has_resolution_incident_finished = True or has_resolution_submitted_payout = True)
- accepted_values:
values:
- false
where: (is_booking_flagged_as_risk is null)
- name: has_resolution_incident_finished
data_type: boolean
description: |
Flag to identify if the booking has a Resolution Incident that has been
finished or not. This means that the resolution has been resolved and
no further action is required.
If True, then the booking has at least one Resolution Incident that has
been finished.
If False, then the booking has no Resolution Incidents that have been
finished - but this does not guarantee that the booking might have a
resolution incident ongoing or in the future.
data_tests:
- not_null
- name: has_resolution_submitted_payout
data_type: boolean
description: |
Flag to identify if the booking has a Resolution Incident that has
submitted a resolution payout or not.
If True, then the booking has at least one Resolution Incident that has
submitted a payout.
If False, then the booking has no Resolution Incidents that have
submitted a payout - but this does not guarantee that the booking might
have a resolution incident with a payout in the future.
data_tests:
- not_null
- name: resolution_submitted_payout_amount_in_gbp
data_type: decimal
description: |
Amount in GBP that has been submitted as a payout for the
Resolution Incident. This is only populated if has_resolution_submitted_payout
is True.
It can be null if the booking has no Resolution Incidents or if the
Resolution Incident has not submitted a payout.
data_tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: true
- is_null:
where: "has_resolution_submitted_payout = False"