counterweight/data-dwh-dbt-project
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merged PR 3578: First version of int_core__booking_service_detail

Browse source 
# Description

Minor Changes:
* Computation in staging of `is_default_service` column for Basic Screening. This includes `stg_core__product_service`, `stg_core__protection_plan` and the schema entry.
* Propagation of `is_default_service` to intermediate. This includes `int_core__protection_plan_to_price`, `int_core__product_service_to_price` and the schema entry.
* Adapting a small data test to include a new case in `stg_core__booking_view_to_service` schema entry.

Main Changes:
* New fields and re-order existing ones on `int_core__booking_to_service`.
* Changed materialisation of `int_core__booking_to_service` to a view, since the new model `int_core__booking_service_detail` is materialised as table.

New Model:
* First version of `int_core__booking_service_detail`. This model includes the current state of a Service that is applied within a Booking. It includes many Booking attributes, Service attributes as well as some meta data fields regarding the business scope and type, according to the [New Pricing](https://www.notion.so/knowyourguest-superhog/All-Services-Definitions-333fba1fbed54715b6b1c3b5133f0d4f) documentation.
* This is one of the 3 Booking models that I propose to create for New Pricing. You can read about the others [here](https://www.notion.so/knowyourguest-superhog/Services-and-Revenue-modelling-1420446ff9c980118e0cfffa7c41f369), although I'll probably won't do them until New Dash is finalised on tech side.
* This current model should suffice to be able to understand which services are applied within a Booking. Notably you will see that this is not enough to compute revenue, as there's some placeholders in place. This is expected.

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #20809
This commit is contained in:
Oriol Roqué Paniagua 2024-11-19 08:24:32 +00:00
parent 30a12f6875
commit f1359bf696
8 changed files with 551 additions and 29 deletions
Download patch file Download diff file Expand all files Collapse all files

188
models/intermediate/core/int_core__booking_service_detail.sql Normal file
View file

@ -0,0 +1,188 @@
{{ config(materialized="table", unique_key=["id_booking_service_detail"]) }}
with
int_core__booking_to_service as (
select * from {{ ref("int_core__booking_to_service") }}
),
stg_core__product_service as (select * from {{ ref("stg_core__product_service") }}),
stg_core__protection_plan as (select * from {{ ref("stg_core__protection_plan") }}),
int_core__product_service_to_price as (
select * from {{ ref("int_core__product_service_to_price") }}
),
int_core__protection_plan_to_price as (
select * from {{ ref("int_core__protection_plan_to_price") }}
),
int_simple_exchange_rates as (select * from {{ ref("int_simple_exchange_rates") }}),
applied_product_services as (
select
bts.id_booking,
bts.id_booking_service_detail,
bts.service_detail_created_at_utc,
bts.service_detail_updated_at_utc,
bts.booking_created_at_utc,
bts.booking_updated_at_utc,
bts.booking_check_in_at_utc,
bts.booking_check_out_at_utc,
'PLATFORM' as service_business_scope,
ps.service_business_type,
'PRODUCT' as service_source,
bts.service_status,
bts.booking_status,
ps.product_service_display_name as service_name,
pstp.payment_type,
pstp.price_base_unit,
pstp.invoicing_trigger,
pstp.currency_code,
pstp.amount_local_curr as service_unit_price_local_curr,
bts.is_missing_host_currency_code as is_missing_currency_code,
bts.is_booking_cancelled,
(not pstp.is_default_service) as is_upgraded_service
from int_core__booking_to_service bts
inner join
stg_core__product_service ps
on bts.id_product_service = ps.id_product_service
left join
int_core__product_service_to_price pstp
on bts.id_product_service = pstp.id_product_service
and bts.host_currency_code = pstp.currency_code
where bts.id_product_service is not null
),
applied_protection_services as (
select
bts.id_booking,
bts.id_booking_service_detail,
bts.service_detail_created_at_utc,
bts.service_detail_updated_at_utc,
bts.booking_created_at_utc,
bts.booking_updated_at_utc,
bts.booking_check_in_at_utc,
bts.booking_check_out_at_utc,
'PLATFORM' as service_business_scope,
'PROTECTION' as service_business_type,
'PROTECTION' as service_source,
bts.service_status,
bts.booking_status,
pp.protection_display_name as service_name,
pptp.payment_type,
pptp.price_base_unit,
pptp.invoicing_trigger,
pptp.currency_code,
pptp.amount_local_curr as service_unit_price_local_curr,
bts.is_missing_host_currency_code as is_missing_currency_code,
bts.is_booking_cancelled,
(not pptp.is_default_service) as is_upgraded_service
from int_core__booking_to_service bts
inner join
stg_core__protection_plan pp
on bts.id_protection_plan = pp.id_protection_plan
left join
int_core__protection_plan_to_price pptp
on bts.id_protection_plan = pptp.id_protection_plan
and bts.host_currency_code = pptp.currency_code
where bts.id_protection_plan is not null
),
applied_other_services as (
select
bts.id_booking,
bts.id_booking_service_detail,
bts.service_detail_created_at_utc,
bts.service_detail_updated_at_utc,
bts.booking_created_at_utc,
bts.booking_updated_at_utc,
bts.booking_check_in_at_utc,
bts.booking_check_out_at_utc,
'PLATFORM' as service_business_scope,
'UNKNOWN' as service_business_type,
'UNKNOWN' as service_source,
bts.service_status,
bts.booking_status,
bts.service_name,
'UNKNOWN' as payment_type,
'UNKNOWN' as price_base_unit,
'UNKNOWN' as invoicing_trigger,
'UNKNOWN' as currency_code,
cast(null as decimal) as service_unit_price_local_curr,
bts.is_missing_host_currency_code as is_missing_currency_code,
bts.is_booking_cancelled,
-- This case shouldn't occur, but the current missing
-- cases are all for Basic Screening which is not an
-- upgraded service.
false as is_upgraded_service
from int_core__booking_to_service bts
where bts.id_protection_plan is null and bts.id_product_service is null
),
service_aggregation as (
select *
from applied_product_services
union all
select *
from applied_protection_services
union all
select *
from applied_other_services
),
local_currency_total_price_computation as (
select
id_booking,
id_booking_service_detail,
service_detail_created_at_utc,
service_detail_updated_at_utc,
booking_created_at_utc,
booking_updated_at_utc,
booking_check_in_at_utc,
booking_check_out_at_utc,
service_business_scope,
service_business_type,
service_source,
service_status,
booking_status,
service_name,
payment_type,
price_base_unit,
invoicing_trigger,
currency_code,
service_unit_price_local_curr,
-- The following is a placeholder for once we have a proper source of truth
cast(null as decimal) as service_total_price_local_curr,
-- The following is a placeholder for once we have a proper source of truth
cast(null as date) as service_charge_date_utc,
is_missing_currency_code,
is_booking_cancelled,
coalesce(is_upgraded_service, false) as is_upgraded_service
from service_aggregation
)
select
id_booking,
id_booking_service_detail,
service_detail_created_at_utc,
service_detail_updated_at_utc,
booking_created_at_utc,
booking_updated_at_utc,
booking_check_in_at_utc,
booking_check_out_at_utc,
service_business_scope,
service_business_type,
service_source,
service_status,
booking_status,
service_name,
payment_type,
price_base_unit,
invoicing_trigger,
currency_code,
service_unit_price_local_curr,
service_unit_price_local_curr * rate as service_unit_price_in_gbp,
service_total_price_local_curr,
service_total_price_local_curr * rate as service_total_price_in_gbp,
service_charge_date_utc,
is_missing_currency_code,
is_booking_cancelled,
case
when service_total_price_local_curr > 0 then true else false
end as is_paid_service,
is_upgraded_service
from local_currency_total_price_computation bsd
left join
int_simple_exchange_rates er
on bsd.service_charge_date_utc = er.rate_date_utc
and bsd.currency_code = er.from_currency
and er.to_currency = 'GBP'

18
models/intermediate/core/int_core__booking_to_service.sql
View file

@ -1,10 +1,11 @@
{{ config(materialized="table", unique_key=["id_booking_view_to_service"]) }}
{{ config(materialized="view", unique_key=["id_booking_service_detail"]) }}
with
stg_core__booking_view_to_service as (
select * from {{ ref("stg_core__booking_view_to_service") }}
),
stg_core__booking_view as (select * from {{ ref("stg_core__booking_view") }}),
stg_core__booking as (select * from {{ ref("stg_core__booking") }}),
stg_core__booking_state as (select * from {{ ref("stg_core__booking_state") }}),
int_core__user_host as (select * from {{ ref("int_core__user_host") }})
select
bv.id_booking,
@ -12,20 +13,29 @@ select
bvts.id_verification,
bvts.id_product_service,
bvts.id_protection_plan,
bvts.service_status,
b.id_user_host,
b.id_user_guest,
bvts.service_name,
bvts.service_status,
upper(bs.booking_state) as booking_status,
bvts.service_protection_amount,
bvts.created_at_utc as service_detail_created_at_utc,
bvts.updated_at_utc as service_detail_updated_at_utc,
b.created_at_utc as booking_created_at_utc,
b.updated_at_utc as booking_updated_at_utc,
b.check_in_at_utc as booking_check_in_at_utc,
b.check_out_at_utc as booking_check_out_at_utc,
b.id_user_host,
uh.account_currency_iso4217 as host_currency_code,
case
when uh.account_currency_iso4217 is null then true else false
end as is_missing_host_currency_code
end as is_missing_host_currency_code,
case
when upper(bs.booking_state) = {{ var("cancelled_booking_state") }}
then true
else false
end as is_booking_cancelled
from stg_core__booking_view_to_service bvts
inner join stg_core__booking_view bv on bvts.id_booking_view = bv.id_booking_view
left join stg_core__booking b on bv.id_booking = b.id_booking
left join stg_core__booking_state bs on b.id_booking_state = bs.id_booking_state
left join int_core__user_host uh on b.id_user_host = uh.id_user_host

1
models/intermediate/core/int_core__product_service_to_price.sql
View file

@ -15,6 +15,7 @@ select
pstp.product_service_price_name,
ps.product_service_name,
ps.product_service_display_name,
ps.is_default_service,
pstp.starts_at_utc,
pstp.ends_at_utc,
pstp.has_no_end_date,

1
models/intermediate/core/int_core__protection_plan_to_price.sql
View file

@ -15,6 +15,7 @@ select
pptp.protection_plan_price_name,
pp.protection_name,
pp.protection_display_name,
pp.is_default_service,
pptp.starts_at_utc,
pptp.ends_at_utc,
pptp.has_no_end_date,

339
models/intermediate/core/schema.yml
View file

@ -2087,7 +2087,7 @@ models:
- name: dwh_extracted_at
data_type: timestamp
description: |
Timestamp of when this row was ingested from the Backend to the DWH.
Timestamp of when this record was ingested from the Backend to the DWH.
- name: int_core__booking_to_product_bundle
description: |
@ -2155,27 +2155,27 @@ models:
- name: created_at_utc
data_type: timestamp
description: |
Timestamp of when this Booking to Product Bundle row was created in the Backend.
Timestamp of when this Booking to Product Bundle record was created in the Backend.
- name: created_date_utc
data_type: date
description: |
Date of when this Booking to Product Bundle row was created in the Backend.
Date of when this Booking to Product Bundle record was created in the Backend.
- name: updated_at_utc
data_type: timestamp
description: |
Timestamp of when this Booking to Product Bundle row was last updated in the Backend.
Timestamp of when this Booking to Product Bundle record was last updated in the Backend.
- name: updated_date_utc
data_type: date
description: |
Date of when this Booking to Product Bundle row was last updated in the Backend.
Date of when this Booking to Product Bundle record was last updated in the Backend.
- name: dwh_extracted_at
data_type: timestamp with timezone
description: |
Timestamp of when this row was ingested from the Backend to the DWH.
Timestamp of when this record was ingested from the Backend to the DWH.
- name: int_core__accommodation_to_product_bundle
description: |
@ -2288,27 +2288,27 @@ models:
- name: created_at_utc
data_type: timestamp
description: |
Timestamp of when this Accommodation to Product Bundle row was created in the Backend.
Timestamp of when this Accommodation to Product Bundle record was created in the Backend.
- name: created_date_utc
data_type: date
description: |
Date of when this Accommodation to Product Bundle row was created in the Backend.
Date of when this Accommodation to Product Bundle record was created in the Backend.
- name: updated_at_utc
data_type: timestamp
description: |
Timestamp of when this Accommodation to Product Bundle row was last updated in the Backend.
Timestamp of when this Accommodation to Product Bundle record was last updated in the Backend.
- name: updated_date_utc
data_type: date
description: |
Date of when this Accommodation to Product Bundle row was last updated in the Backend.
Date of when this Accommodation to Product Bundle record was last updated in the Backend.
- name: dwh_extracted_at
data_type: timestamp with timezone
description: |
Timestamp of when this row was ingested from the Backend to the DWH.
Timestamp of when this record was ingested from the Backend to the DWH.
- name: int_core__new_dash_user_overview
description: |
@ -2639,7 +2639,7 @@ models:
columns:
- name: id_deal
data_type: character varying
description: The unique ID for the Deal. One row per id_deal.
description: The unique ID for the Deal. One record per id_deal.
tests:
- not_null
- unique
@ -2914,6 +2914,13 @@ models:
data_type: string
description: |
The name of the product service, for example "Waiver Pro".
- name: is_default_service
data_type: boolean
description: |
Flag that determines if the service is a default one (True)
or an upgraded service (False).
tests:
- not_null
- name: starts_at_utc
data_type: timestamp
description: |
@ -3033,10 +3040,17 @@ models:
data_type: string
description: |
The name given to a protection plan to price, for example "ProtectionPlus Default Price".
- name: pprotection_plan_name
- name: protection_plan_name
data_type: string
description: |
The name of the protection plan, for example "ProtectionPlus".
- name: is_default_service
data_type: boolean
description: |
Flag that determines if the service is a default one (True)
or an upgraded service (False).
tests:
- not_null
- name: starts_at_utc
data_type: timestamp
description: |
@ -3208,6 +3222,26 @@ models:
The identifier of the protection plan, aka protection service. It acts as
Foreign Key to Product Plan table. It can be null.
- name: id_user_host
data_type: string
description: |
Unique identifier of the user that acts as a Host.
tests:
- not_null
- name: id_user_guest
data_type: string
description: |
Unique identifier of the user that acts as a Guest.
Can be null if Superhog does not interact with the Guest.
- name: service_name
data_type: string
description: |
The name of the service applied within the booking. Cannot be null.
tests:
- not_null
- name: service_status
data_type: string
description: |
@ -3215,10 +3249,10 @@ models:
tests:
- not_null
- name: service_name
- name: booking_status
data_type: string
description: |
The name of the service applied within the booking. Cannot be null.
The status of the overall booking. Cannot be null.
tests:
- not_null
@ -3233,21 +3267,28 @@ models:
- name: service_detail_created_at_utc
data_type: timestamp
description: |
Timestamp of when this Service row was created in the Backend.
Timestamp of when this Service record was created in the Backend.
tests:
- not_null
- name: service_detail_updated_at_utc
data_type: timestamp
description: |
Timestamp of when this Service row was last updated in the Backend.
Timestamp of when this Service record was last updated in the Backend.
tests:
- not_null
- name: booking_created_at_utc
data_type: timestamp
description: |
Timestamp of when the Booking row was created in the Backend.
Timestamp of when the Booking record was created in the Backend.
tests:
- not_null
- name: booking_updated_at_utc
data_type: timestamp
description: |
Timestamp of when the Booking record was last updated in the Backend.
tests:
- not_null
@ -3265,13 +3306,6 @@ models:
tests:
- not_null
- name: id_user_host
data_type: string
description: |
Unique identifier of the user that acts as Host.
tests:
- not_null
- name: host_currency_code
data_type: string
description: |
@ -3282,3 +3316,258 @@ models:
data_type: boolean
description: |
Flag to identify if the host is missing the currency code.
- name: is_booking_cancelled
data_type: boolean
description: |
Flag to identify if the booking has been cancelled or not.
- name: int_core__booking_service_detail
description: |
This model contains enriched information of the services that are applied within
a Booking. Specifically, contains both Booking and Services attributes, as well
as the unit and total prices at this specific moment in time. In other words,
it's the snapshot of the current status of the services applied to a Booking.
It's a subset of all bookings since it only applies to bookings that come from
hosts that have been migrated into the New Dash or New Pricing.
tests:
- dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B:
column_A: booking_check_out_at_utc
column_B: booking_check_in_at_utc
or_equal: True
- dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B:
column_A: service_detail_created_at_utc
column_B: booking_created_at_utc
or_equal: True
- dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B:
column_A: service_detail_updated_at_utc
column_B: service_detail_created_at_utc
or_equal: True
columns:
- name: id_booking
data_type: bigint
description: |
The identifier of the booking. Acts as Foreign Key to Booking table.
Cannot be null.
tests:
- not_null
- name: id_booking_service_detail
data_type: bigint
description: |
The identifier of the booking that has a certain service applied.
The same booking can have multiple services applied and would
have distinct id_booking_service_detail.
tests:
- not_null
- unique
- name: service_detail_created_at_utc
data_type: timestamp
description: |
Timestamp of when this Service record was created in the Backend.
tests:
- not_null
- name: service_detail_updated_at_utc
data_type: timestamp
description: |
Timestamp of when this Service record was last updated in the Backend.
tests:
- not_null
- name: booking_created_at_utc
data_type: timestamp
description: |
Timestamp of when the Booking record was created in the Backend.
tests:
- not_null
- name: booking_updated_at_utc
data_type: timestamp
description: |
Timestamp of when the Booking record was last updated in the Backend.
tests:
- not_null
- name: booking_check_in_at_utc
data_type: timestamp
description: |
Timestamp of the Check-in of the Booking.
tests:
- not_null
- name: booking_check_out_at_utc
data_type: timestamp
description: |
Timestamp of the Check-out of the Booking.
tests:
- not_null
- name: service_business_scope
data_type: string
description: |
Identifies the main business scope (Platform, Guest Products, APIs)
according to New Pricing documentation. Cannot be null.
tests:
- not_null
- accepted_values:
values:
- PLATFORM
- name: service_business_type
data_type: string
description: |
Identifies the service type (Screening, Deposit Management, Protection)
according to New Pricing documentation.
Cannot be null.
tests:
- not_null
- accepted_values:
values:
- SCREENING
- PROTECTION
- DEPOSIT_MANAGEMENT
- UNKNOWN
- name: service_source
data_type: string
description: |
Identifies the source of the information used (Product or Protection
based on how backend is modelled).
tests:
- not_null
- accepted_values:
values:
- PRODUCT
- PROTECTION
- UNKNOWN
- name: service_status
data_type: string
description: |
The status of the service applied within the booking. Cannot be null.
tests:
- not_null
- name: booking_status
data_type: string
description: |
The status of the overall booking. Cannot be null.
tests:
- not_null
- name: service_name
data_type: string
description: |
The name of the service applied within the booking. Cannot be null.
tests:
- not_null
- name: payment_type
data_type: string
description: |
Identifies if the service price unit is an actual amount or a percentage
of another value. It can be null if the host currency is not populated.
tests:
- accepted_values:
values:
- "AMOUNT"
- "PERCENTAGE"
- "UNKNOWN"
- name: price_base_unit
data_type: string
description: |
Identifies if the service price unit needs to be applied per booking or
per number of nights between check-in and check-out.
It can be null if the host currency is not populated.
tests:
- accepted_values:
values:
- "PER BOOKING"
- "PER NIGHT"
- "UNKNOWN"
- name: invoicing_trigger
data_type: string
description: |
Identifies the moment in time in which this service needs to be charged.
It can be null if the host currency is not populated.
tests:
- accepted_values:
values:
- "PRE-BOOKING"
- "POST-CHECKOUT"
- "AT WAIVER PAYMENT"
- "AT DEPOSIT PAYMENT"
- "N/A"
- "UNKNOWN"
- name: currency_code
data_type: string
description: |
Iso 4217 currency code that applies to the Service within the Booking.
It can be null.
- name: service_unit_price_local_curr
data_type: decimal
description: |
Identifies the service unit price in the local currency. Can be null.
- name: service_unit_price_in_gbp
data_type: decimal
description: |
Identifies the service unit price converted to GBP with the rate
of the date of charge. Can be null. Can vary over time until the
charge date due to the currency rate estimation in the future.
- name: service_total_price_local_curr
data_type: decimal
description: |
Identifies the current total price of that service in a given
booking in the local currency. Can be null. Can vary over time
depending on the service status, payments, etc.
- name: service_total_price_in_gbp
data_type: decimal
description: |
Identifies the current total price of that service in a given
booking converted in GBP. Can be null. Can vary over time depending
on the service status, payments, etc, as well as it can vary over
time until the charge date due to the currency rate estimation in
the future.
- name: service_charge_date_utc
data_type: date
description: |
Identifies the moment in time in which the service is charged.
- name: is_missing_currency_code
data_type: boolean
description: |
Flag to identify if the applied service is missing the currency code.
tests:
- not_null
- name: is_booking_cancelled
data_type: boolean
description: |
Flag to identify if the booking has been cancelled or not.
tests:
- not_null
- name: is_paid_service
data_type: boolean
description: |
Flag to identify it the service total price is strictly
greater than 0 or not.
tests:
- not_null
- name: is_upgraded_service
data_type: boolean
description: |
Flag to identify if the service is an upgraded version,
meaning, it's not a Basic Screening.
tests:
- not_null

17
models/staging/core/schema.yml
View file

@ -567,6 +567,15 @@ models:
values:
- "SCREENING"
- "DEPOSIT_MANAGEMENT"
- name: is_default_service
data_type: boolean
description: |
Flag that determines if the service is a default one (True)
or an upgraded service (False).
tests:
- not_null
- name: dwh_extracted_at_utc
data_type: timestamp
description: |
@ -759,6 +768,13 @@ models:
data_type: date
description: |
Date of when this product protection plan was last updated.
- name: is_default_service
data_type: boolean
description: |
Flag that determines if the service is a default one (True)
or an upgraded service (False).
tests:
- not_null
- name: dwh_extracted_at_utc
data_type: timestamp
description: |
@ -1050,6 +1066,7 @@ models:
- "REJECTED"
- "FLAGGED"
- "NOCHECKS"
- "NOT PAID"
- "-"
- name: created_at_utc
data_type: timestamp

8
models/staging/core/stg_core__product_service.sql
View file

@ -41,5 +41,13 @@ select
product_service_display_name,
product_service_description,
product_service_binary_tier,
case
when
id_product_service in (
1 -- 'BASIC SCREENING'
)
then true
else false
end as is_default_service,
dwh_extracted_at_utc
from stg_core__product_service

8
models/staging/core/stg_core__protection_plan.sql
View file

@ -43,6 +43,14 @@ select
pp.created_date_utc,
pp.updated_at_utc,
pp.updated_date_utc,
case
when
id_protection_plan in (
1 -- 'BASIC SCREENING'
)
then true
else false
end as is_default_service,
pp.dwh_extracted_at_utc
from raw_protection_plan pp
left join raw_protection p on pp.id_protection = p.id