counterweight/data-dwh-dbt-project
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merged PR 5366: Adds Incident data to Booking Summary

Browse source 
# Description

This PR moves the categorisation of Bookings with Incidents from the flagging specific model to Booking Summary, effectively keeping this at Booking ID level.

This also handles the direct dependency with the flagging categorisation model.

I also improved the documentation and test coverage on the Booking Summary as it's becoming more and more central to many areas.

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #30676
This commit is contained in:
Oriol Roqué Paniagua 2025-06-02 15:56:54 +00:00
parent 92b144d78e
commit affeac612a
3 changed files with 453 additions and 258 deletions
Download patch file Download diff file Expand all files Collapse all files

435
models/intermediate/cross/int_booking_summary.sql
View file

@ -1,3 +1,21 @@
{% set risk_booking_status = ("NOTAPPROVED", "FLAGGED") %}
{% set no_risk_booking_status = ("APPROVED", "NOFLAGS") %}
{% set incident_duplicated_status = "CLOSED - DUPLICATE" %}
{% set incident_finished_status = (
"RESOLVED",
"RESOLVED EXCEPTION",
"CLOSED - NO REPLY",
"CLOSED - OTHER",
"CLOSED - WAIVER CR",
"CLOSED - LATE REPORT",
"CLOSED - NOT COVERED",
"CLOSED - NOT LIABLE",
"CLOSED HOST REQUEST",
"CLOSED - NOT APPROVED",
"CLOSED - THIRD PARTY",
) %}
{% set days_from_checkout_to_completion = 14 %}
{{ config(materialized="table", unique_key=["id_booking"]) }}
with
int_core__booking_to_service as (
@ -5,172 +23,257 @@ with
),
int_core__booking_service_detail as (
select * from {{ ref("int_core__booking_service_detail") }}
)
select
bts.id_booking,
bts.id_verification_request,
bts.id_accommodation,
bts.id_user_product_bundle,
bts.id_deal,
bts.id_user_host,
bts.id_user_guest,
bts.booking_status,
bts.program_name,
bts.booking_created_at_utc,
bts.booking_created_date_utc,
bts.booking_updated_at_utc,
bts.booking_updated_date_utc,
bts.booking_check_in_at_utc,
bts.booking_check_in_date_utc,
bts.booking_check_out_at_utc,
bts.booking_check_out_date_utc,
bts.booking_number_of_nights,
bts.host_currency_code,
bts.is_user_in_new_dash,
bts.new_dash_version,
bts.user_in_new_dash_since_timestamp_at_utc,
sum(bsd.service_total_price_in_gbp) as booking_total_price_in_gbp,
min(bsd.service_first_chargeable_date_utc) as service_first_chargeable_date_utc,
max(bsd.service_last_chargeable_date_utc) as service_last_chargeable_date_utc,
min(bsd.service_first_billable_date_utc) as service_first_billable_date_utc,
max(bsd.service_last_billable_date_utc) as service_last_billable_date_utc,
min(bsd.service_detail_created_at_utc) as service_first_created_at_utc,
max(bsd.service_detail_created_at_utc) as service_last_created_at_utc,
max(bsd.service_detail_updated_at_utc) as service_last_updated_at_utc,
count(distinct bsd.id_booking_service_detail) as number_of_applied_services,
count(
distinct case
when bsd.is_paid_service then bsd.id_booking_service_detail else null
end
) as number_of_applied_paid_services,
count(
distinct case
when bsd.is_upgraded_service then bsd.id_booking_service_detail else null
end
) as number_of_applied_upgraded_services,
count(
distinct case
when bsd.is_billable_service then bsd.id_booking_service_detail else null
end
) as number_of_applied_billable_services,
case
when
sum(bsd.service_total_price_in_gbp) > 0
and min(bsd.service_first_chargeable_date_utc) is not null
then true
else false
end as is_booking_chargeable,
case
when
sum(
case
),
int_resolutions__incidents as (
select * from {{ ref("int_resolutions__incidents") }}
),
-- This CTE handles the core backend booking information and aggregates service
-- details.
core_backend_booking_modelling as (
select
bts.id_booking,
bts.id_verification_request,
bts.id_accommodation,
bts.id_user_product_bundle,
bts.id_deal,
bts.id_user_host,
bts.id_user_guest,
bts.booking_status,
bts.program_name,
bts.booking_created_at_utc,
bts.booking_created_date_utc,
bts.booking_updated_at_utc,
bts.booking_updated_date_utc,
bts.booking_check_in_at_utc,
bts.booking_check_in_date_utc,
bts.booking_check_out_at_utc,
bts.booking_check_out_date_utc,
bts.booking_check_out_date_utc
+ {{ days_from_checkout_to_completion }} as booking_completed_date_utc,
bts.booking_number_of_nights,
bts.host_currency_code,
bts.is_user_in_new_dash,
bts.new_dash_version,
bts.user_in_new_dash_since_timestamp_at_utc,
sum(bsd.service_total_price_in_gbp) as booking_total_price_in_gbp,
min(
bsd.service_first_chargeable_date_utc
) as service_first_chargeable_date_utc,
max(
bsd.service_last_chargeable_date_utc
) as service_last_chargeable_date_utc,
min(bsd.service_first_billable_date_utc) as service_first_billable_date_utc,
max(bsd.service_last_billable_date_utc) as service_last_billable_date_utc,
min(bsd.service_detail_created_at_utc) as service_first_created_at_utc,
max(bsd.service_detail_created_at_utc) as service_last_created_at_utc,
max(bsd.service_detail_updated_at_utc) as service_last_updated_at_utc,
count(distinct bsd.id_booking_service_detail) as number_of_applied_services,
count(
distinct case
when bsd.is_paid_service
then bsd.id_booking_service_detail
else null
end
) as number_of_applied_paid_services,
count(
distinct case
when bsd.is_upgraded_service
then bsd.id_booking_service_detail
else null
end
) as number_of_applied_upgraded_services,
count(
distinct case
when bsd.is_billable_service
then bsd.service_total_price_in_gbp
else 0
then bsd.id_booking_service_detail
else null
end
)
> 0
and min(bsd.service_first_billable_date_utc) is not null
then true
else false
end as is_booking_billable,
) as number_of_applied_billable_services,
case
when
sum(bsd.service_total_price_in_gbp) > 0
and min(bsd.service_first_chargeable_date_utc) is not null
then true
else false
end as is_booking_chargeable,
case
when
sum(
case
when bsd.is_billable_service
then bsd.service_total_price_in_gbp
else 0
end
)
> 0
and min(bsd.service_first_billable_date_utc) is not null
then true
else false
end as is_booking_billable,
case
when sum(case when bsd.is_missing_currency_code then 1 else 0 end) > 0
then true
else false
end as is_missing_currency_code_in_service_detail,
case
when sum(case when bsd.is_booking_cancelled then 1 else 0 end) > 0
then true
else false
end as is_booking_cancelled,
case
when bts.id_verification_request is null then false else true
end as has_verification_request,
case
when sum(case when bsd.is_paid_service then 1 else 0 end) > 0
then true
else false
end as has_paid_services,
case
when sum(case when bsd.is_upgraded_service then 1 else 0 end) > 0
then true
else false
end as has_upgraded_services,
case
when sum(case when bsd.is_billable_service then 1 else 0 end) > 0
then true
else false
end as has_billable_services,
case
when
sum(
case
when bsd.service_business_type = 'SCREENING' then 1 else 0
end
)
> 0
then true
else false
end as has_screening_service_business_type,
case
when
sum(
case
when
bsd.service_business_type = 'SCREENING'
and bsd.service_name <> {{ var("default_service") }}
then 1
else 0
end
)
> 0
then true
else false
end as has_upgraded_screening_service_business_type,
case
when
sum(
case
when bsd.service_business_type = 'DEPOSIT_MANAGEMENT'
then 1
else 0
end
)
> 0
then true
else false
end as has_deposit_management_service_business_type,
case
when
sum(
case
when bsd.service_business_type = 'PROTECTION' then 1 else 0
end
)
> 0
then true
else false
end as has_protection_service_business_type,
bts.is_missing_id_deal,
case
when bts.host_currency_code is null then true else false
end as is_missing_host_currency_code,
case
when
(current_date - booking_check_out_date_utc)
> {{ days_from_checkout_to_completion }}
then true
else false
end as is_booking_past_completion_date,
case
when upper(bts.booking_status) in {{ risk_booking_status }}
then true
when upper(bts.booking_status) in {{ no_risk_booking_status }}
then false
else null
end as is_booking_flagged_as_risk
from int_core__booking_to_service bts
inner join
int_core__booking_service_detail bsd
on bts.id_booking_service_detail = bsd.id_booking_service_detail
group by
bts.id_booking,
bts.id_verification_request,
bts.id_accommodation,
bts.id_user_product_bundle,
bts.id_deal,
bts.id_user_host,
bts.id_user_guest,
bts.booking_status,
bts.program_name,
bts.booking_created_at_utc,
bts.booking_created_date_utc,
bts.booking_updated_at_utc,
bts.booking_updated_date_utc,
bts.booking_check_in_at_utc,
bts.booking_check_in_date_utc,
bts.booking_check_out_at_utc,
bts.booking_check_out_date_utc,
bts.booking_number_of_nights,
bts.host_currency_code,
bts.is_missing_id_deal,
bts.is_user_in_new_dash,
bts.new_dash_version,
bts.user_in_new_dash_since_timestamp_at_utc
),
-- This CTE deduplicates resolution incidents by booking ID and aggregates payout
-- amounts.
deduplicated_incidents as (
select
id_booking,
sum(submitted_payout_amount_in_gbp) as submitted_payout_amount_in_gbp,
case
when sum(submitted_payout_amount_in_gbp) > 0 then true else false
end as has_submitted_payout,
case
when
sum(
case
when
upper(current_status_name)
in {{ incident_finished_status }}
then 1
else 0
end
)
> 0
then true
else false
end as is_incident_finished
from int_resolutions__incidents
where upper(current_status_name) != '{{ incident_duplicated_status }}'
group by 1
)
-- This CTE combines the core backend booking information with the deduplicated
-- incidents to provide a comprehensive booking summary.
select
-- Retrieve all previous backend booking modelling fields.
cbbm.*,
-- Retrieve incident resolution fields.
case
when sum(case when bsd.is_missing_currency_code then 1 else 0 end) > 0
then true
else false
end as is_missing_currency_code_in_service_detail,
case
when sum(case when bsd.is_booking_cancelled then 1 else 0 end) > 0
then true
else false
end as is_booking_cancelled,
case
when bts.id_verification_request is null then false else true
end as has_verification_request,
case
when sum(case when bsd.is_paid_service then 1 else 0 end) > 0
then true
else false
end as has_paid_services,
case
when sum(case when bsd.is_upgraded_service then 1 else 0 end) > 0
then true
else false
end as has_upgraded_services,
case
when sum(case when bsd.is_billable_service then 1 else 0 end) > 0
then true
else false
end as has_billable_services,
case
when
sum(case when bsd.service_business_type = 'SCREENING' then 1 else 0 end) > 0
then true
else false
end as has_screening_service_business_type,
case
when
sum(
case
when
bsd.service_business_type = 'SCREENING'
and bsd.service_name <> {{ var("default_service") }}
then 1
else 0
end
)
> 0
then true
else false
end as has_upgraded_screening_service_business_type,
case
when
sum(
case
when bsd.service_business_type = 'DEPOSIT_MANAGEMENT' then 1 else 0
end
)
> 0
then true
else false
end as has_deposit_management_service_business_type,
case
when
sum(case when bsd.service_business_type = 'PROTECTION' then 1 else 0 end)
> 0
then true
else false
end as has_protection_service_business_type,
bts.is_missing_id_deal,
case
when bts.host_currency_code is null then true else false
end as is_missing_host_currency_code
from int_core__booking_to_service bts
inner join
int_core__booking_service_detail bsd
on bts.id_booking_service_detail = bsd.id_booking_service_detail
group by
bts.id_booking,
bts.id_verification_request,
bts.id_accommodation,
bts.id_user_product_bundle,
bts.id_deal,
bts.id_user_host,
bts.id_user_guest,
bts.booking_status,
bts.program_name,
bts.booking_created_at_utc,
bts.booking_created_date_utc,
bts.booking_updated_at_utc,
bts.booking_updated_date_utc,
bts.booking_check_in_at_utc,
bts.booking_check_in_date_utc,
bts.booking_check_out_at_utc,
bts.booking_check_out_date_utc,
bts.booking_number_of_nights,
bts.host_currency_code,
bts.is_missing_id_deal,
bts.is_user_in_new_dash,
bts.new_dash_version,
bts.user_in_new_dash_since_timestamp_at_utc
when di.id_booking is not null then true else false
end as has_resolution_incident,
coalesce(di.is_incident_finished, false) as has_resolution_incident_finished,
coalesce(di.has_submitted_payout, false) as has_resolution_submitted_payout,
submitted_payout_amount_in_gbp as resolution_submitted_payout_amount_in_gbp
from core_backend_booking_modelling cbbm
left join deduplicated_incidents di on cbbm.id_booking = di.id_booking

94
models/intermediate/cross/int_flagging_booking_categorisation.sql
View file

@ -1,71 +1,17 @@
{% set risk_booking_status = ("NOTAPPROVED", "FLAGGED") %}
{% set no_risk_booking_status = ("APPROVED", "NOFLAGS") %}
{% set incident_duplicated_status = "CLOSED - DUPLICATE" %}
{% set incident_finished_status = (
"RESOLVED",
"RESOLVED EXCEPTION",
"CLOSED - NO REPLY",
"CLOSED - OTHER",
"CLOSED - WAIVER CR",
"CLOSED - LATE REPORT",
"CLOSED - NOT COVERED",
"CLOSED - NOT LIABLE",
"CLOSED HOST REQUEST",
"CLOSED - NOT APPROVED",
"CLOSED - THIRD PARTY",
) %}
{% set days_from_checkout_to_completion = 14 %}
{{ config(materialized="table") }}
with
int_booking_summary as (select * from {{ ref("int_booking_summary") }}),
int_resolutions__incidents as (
select * from {{ ref("int_resolutions__incidents") }}
),
-- The same booking can have multiple Incidents
deduplicated_incidents as (
select
id_booking,
sum(submitted_payout_amount_in_gbp) as submitted_payout_amount_in_gbp,
case
when sum(submitted_payout_amount_in_gbp) > 0 then true else false
end as has_submitted_payout,
case
when
sum(
case
when
upper(current_status_name)
in {{ incident_finished_status }}
then 1
else 0
end
)
> 0
then true
else false
end as is_incident_finished
from int_resolutions__incidents
where upper(current_status_name) != '{{ incident_duplicated_status }}'
group by 1
),
new_dash_protected_bookings as (
select
id_booking,
case
when
(current_date - booking_check_out_date_utc)
> {{ days_from_checkout_to_completion }}
then true
else false
end as is_booking_completed,
case
when upper(booking_status) in {{ risk_booking_status }}
then true
when upper(booking_status) in {{ no_risk_booking_status }}
then false
else null
end as is_booking_flagged_as_risk
is_booking_past_completion_date as is_booking_completed,
is_booking_flagged_as_risk,
has_resolution_incident as has_claim,
has_resolution_submitted_payout as has_submitted_payout,
has_resolution_incident_finished as is_incident_finished,
coalesce(
resolution_submitted_payout_amount_in_gbp, 0
) as submitted_payout_amount_in_gbp
from int_booking_summary
where
-- Bookings from New Dash users with Id Deal
@ -76,25 +22,9 @@ with
has_protection_service_business_type
or has_deposit_management_service_business_type
)
-- Bookings with relevant status (i.e. not cancelled, not pending)
and (
upper(booking_status) in {{ risk_booking_status }}
or upper(booking_status) in {{ no_risk_booking_status }}
)
),
bookings_with_incidents as (
select
ndpb.id_booking,
ndpb.is_booking_completed,
ndpb.is_booking_flagged_as_risk,
case when di.id_booking is not null then true else false end as has_claim,
coalesce(di.has_submitted_payout, false) as has_submitted_payout,
coalesce(di.is_incident_finished, false) as is_incident_finished,
coalesce(
submitted_payout_amount_in_gbp, 0
) as submitted_payout_amount_in_gbp
from new_dash_protected_bookings ndpb
left join deduplicated_incidents di on ndpb.id_booking = di.id_booking
-- Bookings with flagging categorisation (this excludes
-- Cancelled/Incomplete Information/Rejected bookings)
and is_booking_flagged_as_risk is not null
)
select
-- High Level Bookings --
@ -275,4 +205,4 @@ select
and is_incident_finished
) as completed_no_risk_with_submitted_payout_amount_paid_in_gbp
from bookings_with_incidents
from new_dash_protected_bookings

182
models/intermediate/cross/schema.yml
View file

@ -3406,20 +3406,42 @@ models:
- name: int_booking_summary
description: |
This model contains enriched information aggregated at Booking level regarding
the services that are applied within a Booking.
Specifically, contains both Booking and Services attributes (aggregated), as well
as the total price in GBP at this specific moment in time. In other words,
it's the snapshot of the current status of the Booking.
It's a subset of all bookings since it only applies to bookings that come from
hosts that have been migrated into the New Dash or New Pricing.
This model contains enriched information aggregated at Booking level under New Pricing
regarding the services that are applied within a Booking.
If you want to retrieve only New Dash Bookings, you will need to apply the following
filters:
Specifically, contains both Booking and Services attributes (aggregated), as well
as the total price in GBP at this specific moment in time. This model also contains
information on the potential Resolution Incidents linked to a Booking, as long as
these appear in the Resolution Center.
In other words, it's the snapshot of the current status of the Booking.
Be aware that this model only contains a subset of all bookings. It only applies to
1) Platform (Dashboard) bookings, so APIs are excluded and,
2) that come from hosts that have been migrated into the New Pricing.
In essence, that the Booking is linked to a User Product Bundle.
It's likely that you are just interested in retrieveing only New Dash Bookings. For this,
you will need to apply the following filters:
- is_user_in_new_dash = True
- is_missing_id_deal = False
For Data-Driven Flagging purposes, you're likely going to be interested in applying all or
part of the following filters:
1. Bookings from New Dash users with Id Deal
- is_user_in_new_dash = True
- is_missing_id_deal = False
2. Protected Bookings with a Protection or a Deposit Management service
- has_protection_service_business_type or has_deposit_management_service_business_type
3. Bookings with flagging categorisation (this excludes Cancelled/Incomplete/Rejected bookings)
- is_booking_flagged_as_risk is not null
data_tests:
- dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B:
column_A: booking_completed_date_utc
column_B: booking_check_out_at_utc
or_equal: False
- dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B:
column_A: booking_check_out_at_utc
column_B: booking_check_in_at_utc
@ -3490,6 +3512,15 @@ models:
The current status of the booking. Cannot be null.
data_tests:
- not_null
- accepted_values:
values:
- APPROVED
- NOFLAGS
- NOTAPPROVED
- FLAGGED
- CANCELLED
- INCOMPLETEINFORMATION
- REJECTED
- name: program_name
data_type: string
@ -3562,6 +3593,16 @@ models:
data_tests:
- not_null
- name: booking_completed_date_utc
data_type: date
description: |
Date of the Booking completion. This is assumed to be 14 days
after Check-out date, and it is used to determine the limit in
which the Host can raise a Resolution Incident regarding this
booking.
data_tests:
- not_null
- name: booking_number_of_nights
data_type: integer
description: |
@ -3661,24 +3702,51 @@ models:
data_type: integer
description: |
Total number of Services applied to this Booking.
data_tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: true
- name: number_of_applied_paid_services
data_type: integer
description: |
Total number of Services that require a monetary
income to Superhog applied to this Booking.
data_tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
max_value: number_of_applied_upgraded_services
strictly: false
- name: number_of_applied_upgraded_services
data_type: integer
description: |
Total number of Services different from Basic Screening
applied to this Booking.
data_tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
max_value: number_of_applied_services
strictly: false
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: true
where: (has_protection_service_business_type = True or has_deposit_management_service_business_type = True)
- name: number_of_applied_billable_services
data_type: integer
description: |
Total number of Services that require an invoice to the Host
to the Host, that are applied to this Booking.
that are applied to this Booking.
data_tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
max_value: number_of_applied_upgraded_services
strictly: false
- name: is_booking_chargeable
data_type: boolean
@ -3784,3 +3852,97 @@ models:
description: |
Flag to identify if the Host for this booking is missing
the currency code or not.
- name: is_booking_past_completion_date
data_type: boolean
description: |
Flag to identify if the booking is past the completion date or not.
This is used to determine if the booking can be considered as completed
or not, and therefore, if it can be used for further risk-assessment
analysis.
data_tests:
- not_null
- name: is_booking_flagged_as_risk
data_type: boolean
description: |
Flag to identify if the booking has been flagged as risk or not.
This is based on the Booking Status and will return:
- True if the status is "NOTAPPROVED" or "FLAGGED"
- False if the status is "APPROVED" or "NOFLAGS"
- Null otherwise, such as "CANCELLED", "INCOMPLETEINFORMATION" or "REJECTED"
data_tests:
- accepted_values:
values:
- true
where: (upper(booking_status) in ('NOTAPPROVED', 'FLAGGED'))
- accepted_values:
values:
- false
where: (upper(booking_status) in ('APPROVED', 'NOFLAGS'))
- dbt_expectations.expect_column_values_to_be_null:
where: (upper(booking_status) not in ('NOTAPPROVED', 'FLAGGED', 'APPROVED', 'NOFLAGS'))
- name: has_resolution_incident
data_type: boolean
description: |
Flag to identify if the booking has a Resolution Incident or not. This might
imply that the resolution is in progress or that has been resolved, so it's not
necessarily finished.
If True, then the booking has at least one Resolution Incident.
If False, then the booking has no Resolution Incidents - but this does not
guarantee that the booking might have a resolution incident in the future.
data_tests:
- not_null
- accepted_values:
values:
- true
where: (has_resolution_incident_finished = True or has_resolution_submitted_payout = True)
- accepted_values:
values:
- false
where: (is_booking_flagged_as_risk is null)
- name: has_resolution_incident_finished
data_type: boolean
description: |
Flag to identify if the booking has a Resolution Incident that has been
finished or not. This means that the resolution has been resolved and
no further action is required.
If True, then the booking has at least one Resolution Incident that has
been finished.
If False, then the booking has no Resolution Incidents that have been
finished - but this does not guarantee that the booking might have a
resolution incident ongoing or in the future.
data_tests:
- not_null
- name: has_resolution_submitted_payout
data_type: boolean
description: |
Flag to identify if the booking has a Resolution Incident that has
submitted a resolution payout or not.
If True, then the booking has at least one Resolution Incident that has
submitted a payout.
If False, then the booking has no Resolution Incidents that have
submitted a payout - but this does not guarantee that the booking might
have a resolution incident with a payout in the future.
data_tests:
- not_null
- name: resolution_submitted_payout_amount_in_gbp
data_type: decimal
description: |
Amount in GBP that has been submitted as a payout for the
Resolution Incident. This is only populated if has_resolution_submitted_payout
is True.
It can be null if the booking has no Resolution Incidents or if the
Resolution Incident has not submitted a payout.
data_tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: true
- is_null:
where: "has_resolution_submitted_payout = False"