counterweight/data-dwh-dbt-project
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merged PR 2583: Invoicing metrics per customer segmentation

Browse source 
# Description

Silly change:

* Modifies `int_core__mtd_guest_payments_metrics` to apply the proper key on date, dimension and dimension_value.
* -> *The weird thing is that the previous dbt test I run worked well. Is it possible that the configuration in the schema file prevails on top of the model configuration? I thought it was the other way around...*

Main changes:

* Modifies `int_xero__mtd_invoicing_metrics` to include the customer segmentation based on listings.
* `schema.yaml` is also affected including new fields and tests
* Added the macro to retrieve the production dimension in `int_core__mtd_vs_previous_year_metrics` to avoid propagating this upwards and messing up with the data display.

Overall, follows a similar strategy as we did for Booking, Guest Journey, Deal, Accommodation and Guest Payments metrics. For reference, [here's the previous PR on Guest Payments](https://guardhog.visualstudio.com/Data/_git/data-dwh-dbt-project/pullrequest/2580).

This is the last PR on the source models for KPIs. Will follow: propagation + exposure

# Checklist

- [X] The edited models and dependants run properly with production data.
* **Important note**: this segmentation provides null values for all API-related KPIs. Makes sense, since the 4 deal id we have for APIs do NOT have, or have had, a listing linked to them. I'd say it's not a blocker.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #19325
This commit is contained in:
Oriol Roqué Paniagua 2024-08-20 12:05:53 +00:00
parent 68f490f9fa
commit 80abac494a
4 changed files with 183 additions and 97 deletions
Download patch file Download diff file Expand all files Collapse all files

2
models/intermediate/core/int_core__mtd_guest_payments_metrics.sql
View file

@ -8,7 +8,7 @@ This model provides Month-To-Date (MTD) based on Guest Revenue metrics.
{% set waiver_fees_verification_payment_type_items = "('WAIVER')" %} {% set waiver_fees_verification_payment_type_items = "('WAIVER')" %}
{% set checkin_cover_fees_verification_payment_type_items = "('CHECKINCOVER')" %} {% set checkin_cover_fees_verification_payment_type_items = "('CHECKINCOVER')" %}
{{ config(materialized="table", unique_key="date") }} {{ config(materialized="table", unique_key=["date", "dimension", "dimension_value"]) }}
with with
int_core__verification_payments as ( int_core__verification_payments as (
select * from {{ ref("int_core__verification_payments") }} select * from {{ ref("int_core__verification_payments") }}

1
models/intermediate/cross/int_mtd_vs_previous_year_metrics.sql
View file

@ -29,6 +29,7 @@ with
), ),
int_xero__mtd_invoicing_metrics as ( int_xero__mtd_invoicing_metrics as (
select * from {{ ref("int_xero__mtd_invoicing_metrics") }} select * from {{ ref("int_xero__mtd_invoicing_metrics") }}
where dimension in ({{ production_dimensions }})
), ),
int_dates_mtd as (select * from {{ ref("int_dates_mtd") }}), int_dates_mtd as (select * from {{ ref("int_dates_mtd") }}),

254
models/intermediate/xero/int_xero__mtd_invoicing_metrics.sql
View file

@ -4,6 +4,8 @@ invoicing metrics. This includes Operator (Host) fees, waiver payments,
APIs and Host Resolutions. APIs and Host Resolutions.
*/ */
{% set dimensions = get_kpi_dimensions() %}
{% set relevant_document_statuses = "('PAID', 'AUTHORISED')" %} {% set relevant_document_statuses = "('PAID', 'AUTHORISED')" %}
{% set booking_fee_items = "('EU BOOKING FEE','ZAR BOOKINGS','BOOKING FEE - NON-UK','USD BOOKINGS','CAD BOOKINGS','BOOKING FEE - UK','AUD BOOKINGS')" %} {% set booking_fee_items = "('EU BOOKING FEE','ZAR BOOKINGS','BOOKING FEE - NON-UK','USD BOOKINGS','CAD BOOKINGS','BOOKING FEE - UK','AUD BOOKINGS')" %}
{% set listing_fee_items = "('USD LISTINGS','LISTING FEE - NON UK','ZAR LISTINGS','CAD LISTINGS','LISTING FEE - UK','AUD LISTINGS','EU LISTINGS')" %} {% set listing_fee_items = "('USD LISTINGS','LISTING FEE - NON UK','ZAR LISTINGS','CAD LISTINGS','LISTING FEE - UK','AUD LISTINGS','EU LISTINGS')" %}
@ -13,7 +15,7 @@ APIs and Host Resolutions.
{% set e_deposit_account_name = "('E-DEPOSIT FEES')" %} {% set e_deposit_account_name = "('E-DEPOSIT FEES')" %}
{% set guesty_account_name = "('GUESTY FEES', 'GUESTY ADMINISTRATION FEE')" %} {% set guesty_account_name = "('GUESTY FEES', 'GUESTY ADMINISTRATION FEE')" %}
{{ config(materialized="table", unique_key="date") }} {{ config(materialized="table", unique_key=["date", "dimension", "dimension_value"]) }}
with with
int_xero__sales_denom_mart as ( int_xero__sales_denom_mart as (
select * from {{ ref("int_xero__sales_denom_mart") }} select * from {{ ref("int_xero__sales_denom_mart") }}
@ -25,101 +27,149 @@ with
int_xero__bank_transactions as ( int_xero__bank_transactions as (
select * from {{ ref("int_xero__bank_transactions") }} select * from {{ ref("int_xero__bank_transactions") }}
), ),
int_core__mtd_accommodation_segmentation as (
select * from {{ ref("int_core__mtd_accommodation_segmentation") }}
),
int_dates_mtd_by_dimension as (
select * from {{ ref("int_dates_mtd_by_dimension") }}
),
int_xero__contacts as (select * from {{ ref("int_xero__contacts") }}),
resolution_host_payment as ( resolution_host_payment as (
select {% for dimension in dimensions %}
d.date, select
sum( d.date,
btli.line_amount_wo_taxes_in_gbp {{ dimension.dimension }} as dimension,
) as xero_host_resolution_amount_paid_in_gbp, {{ dimension.dimension_value }} as dimension_value,
count(distinct bt.id_bank_transaction) as xero_host_resolution_payment_count sum(
from int_dates_mtd d btli.line_amount_wo_taxes_in_gbp
inner join ) as xero_host_resolution_amount_paid_in_gbp,
int_xero__bank_transactions bt count(
on date_trunc('month', bt.transaction_date_utc)::date = d.first_day_month distinct bt.id_bank_transaction
and extract(day from bt.transaction_date_utc) <= d.day ) as xero_host_resolution_payment_count
inner join from int_dates_mtd d
int_xero__bank_transaction_line_items btli inner join
on bt.id_bank_transaction = btli.id_bank_transaction int_xero__bank_transactions bt
and upper(btli.account_name) in {{ resolutions_host_payment_account_name }} on date_trunc('month', bt.transaction_date_utc)::date = d.first_day_month
group by 1 and extract(day from bt.transaction_date_utc) <= d.day
inner join
int_xero__bank_transaction_line_items btli
on bt.id_bank_transaction = btli.id_bank_transaction
and upper(btli.account_name)
in {{ resolutions_host_payment_account_name }}
{% if dimension.dimension == "'by_number_of_listings'" %}
inner join int_xero__contacts c on c.id_contact = bt.id_contact
inner join
int_core__mtd_accommodation_segmentation mas
on c.id_deal = mas.id_deal
and d.date = mas.date
{% endif %}
group by 1, 2, 3
{% if not loop.last %}
union all
{% endif %}
{% endfor %}
), ),
apis_net_fees as ( apis_net_fees as (
select {% for dimension in dimensions %}
d.date, select
sum( d.date,
case {{ dimension.dimension }} as dimension,
when upper(sdm.account_name) in {{ e_deposit_account_name }} {{ dimension.dimension_value }} as dimension_value,
then sdm.line_amount_wo_taxes_in_gbp sum(
else 0 case
end when upper(sdm.account_name) in {{ e_deposit_account_name }}
) as xero_e_deposit_net_fees_in_gbp, then sdm.line_amount_wo_taxes_in_gbp
sum( else 0
case end
when upper(sdm.account_name) in {{ guesty_account_name }} ) as xero_e_deposit_net_fees_in_gbp,
then sdm.line_amount_wo_taxes_in_gbp sum(
else 0 case
end when upper(sdm.account_name) in {{ guesty_account_name }}
) as xero_guesty_net_fees_in_gbp then sdm.line_amount_wo_taxes_in_gbp
from int_dates_mtd d else 0
inner join end
int_xero__sales_denom_mart sdm ) as xero_guesty_net_fees_in_gbp
on date_trunc('month', sdm.document_issued_date_utc) = d.first_day_month from int_dates_mtd d
and extract(day from sdm.document_issued_date_utc) <= d.day inner join
where int_xero__sales_denom_mart sdm
upper(sdm.document_status) in {{ relevant_document_statuses }} on date_trunc('month', sdm.document_issued_date_utc) = d.first_day_month
and ( and extract(day from sdm.document_issued_date_utc) <= d.day
upper(sdm.account_name) in {{ e_deposit_account_name }} {% if dimension.dimension == "'by_number_of_listings'" %}
or upper(sdm.account_name) in {{ guesty_account_name }} inner join
) int_core__mtd_accommodation_segmentation mas
group by 1 on sdm.id_deal = mas.id_deal
and d.date = mas.date
{% endif %}
where
upper(sdm.document_status) in {{ relevant_document_statuses }}
and (
upper(sdm.account_name) in {{ e_deposit_account_name }}
or upper(sdm.account_name) in {{ guesty_account_name }}
)
group by 1, 2, 3
{% if not loop.last %}
union all
{% endif %}
{% endfor %}
), ),
host_net_fees as ( host_net_fees as (
select {% for dimension in dimensions %}
d.date, select
sum( d.date,
case {{ dimension.dimension }} as dimension,
when upper(sdm.item_code) in {{ booking_fee_items }} {{ dimension.dimension_value }} as dimension_value,
then sdm.line_amount_wo_taxes_in_gbp sum(
else 0 case
end when upper(sdm.item_code) in {{ booking_fee_items }}
) as xero_booking_net_fees_in_gbp, then sdm.line_amount_wo_taxes_in_gbp
sum( else 0
case end
when upper(sdm.item_code) in {{ listing_fee_items }} ) as xero_booking_net_fees_in_gbp,
then sdm.line_amount_wo_taxes_in_gbp sum(
else 0 case
end when upper(sdm.item_code) in {{ listing_fee_items }}
) as xero_listing_net_fees_in_gbp, then sdm.line_amount_wo_taxes_in_gbp
sum( else 0
case end
when upper(sdm.item_code) in {{ verification_fee_items }} ) as xero_listing_net_fees_in_gbp,
then sdm.line_amount_wo_taxes_in_gbp sum(
else 0 case
end when upper(sdm.item_code) in {{ verification_fee_items }}
) as xero_verification_net_fees_in_gbp, then sdm.line_amount_wo_taxes_in_gbp
sum( else 0
case end
when upper(sdm.item_code) in {{ waiver_items }} ) as xero_verification_net_fees_in_gbp,
then sdm.line_amount_wo_taxes_in_gbp sum(
else 0 case
end when upper(sdm.item_code) in {{ waiver_items }}
) as xero_waiver_paid_back_to_host_in_gbp then sdm.line_amount_wo_taxes_in_gbp
from int_dates_mtd d else 0
inner join end
int_xero__sales_denom_mart sdm ) as xero_waiver_paid_back_to_host_in_gbp
on date_trunc('month', sdm.document_issued_date_utc) = d.first_day_month from int_dates_mtd d
and extract(day from sdm.document_issued_date_utc) <= d.day inner join
where int_xero__sales_denom_mart sdm
upper(sdm.document_status) in {{ relevant_document_statuses }} on date_trunc('month', sdm.document_issued_date_utc) = d.first_day_month
and ( and extract(day from sdm.document_issued_date_utc) <= d.day
upper(sdm.item_code) in {{ booking_fee_items }} {% if dimension.dimension == "'by_number_of_listings'" %}
or upper(sdm.item_code) in {{ listing_fee_items }} inner join
or upper(sdm.item_code) in {{ verification_fee_items }} int_core__mtd_accommodation_segmentation mas
or upper(sdm.item_code) in {{ waiver_items }} on sdm.id_deal = mas.id_deal
) and d.date = mas.date
group by 1 {% endif %}
where
upper(sdm.document_status) in {{ relevant_document_statuses }}
and (
upper(sdm.item_code) in {{ booking_fee_items }}
or upper(sdm.item_code) in {{ listing_fee_items }}
or upper(sdm.item_code) in {{ verification_fee_items }}
or upper(sdm.item_code) in {{ waiver_items }}
)
group by 1, 2, 3
{% if not loop.last %}
union all
{% endif %}
{% endfor %}
) )
-- Final aggregation of subqueries -- -- Final aggregation of subqueries --
select select
@ -127,6 +177,8 @@ select
d.month, d.month,
d.day, d.day,
d.date, d.date,
d.dimension,
d.dimension_value,
d.is_end_of_month, d.is_end_of_month,
d.is_current_month, d.is_current_month,
-- HOST/OPERATOR -- -- HOST/OPERATOR --
@ -152,7 +204,19 @@ select
-- HOST RESOLUTIONS -- -- HOST RESOLUTIONS --
rhp.xero_host_resolution_amount_paid_in_gbp, rhp.xero_host_resolution_amount_paid_in_gbp,
rhp.xero_host_resolution_payment_count rhp.xero_host_resolution_payment_count
from int_dates_mtd d from int_dates_mtd_by_dimension d
left join resolution_host_payment rhp on rhp.date = d.date left join
left join apis_net_fees anf on anf.date = d.date resolution_host_payment rhp
left join host_net_fees hnf on hnf.date = d.date on rhp.date = d.date
and rhp.dimension = d.dimension
and rhp.dimension_value = d.dimension_value
left join
apis_net_fees anf
on anf.date = d.date
and anf.dimension = d.dimension
and anf.dimension_value = d.dimension_value
left join
host_net_fees hnf
on hnf.date = d.date
and hnf.dimension = d.dimension
and hnf.dimension_value = d.dimension_value

23
models/intermediate/xero/schema.yaml
View file

@ -291,10 +291,31 @@ models:
It's used for the business KPIs. Data is aggregated at the last day of the month and in the It's used for the business KPIs. Data is aggregated at the last day of the month and in the
days necessary for the Month-to-Date computation of the current month. days necessary for the Month-to-Date computation of the current month.
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- date
- dimension
- dimension_value
columns: columns:
- name: date - name: date
data_type: date data_type: date
description: The date for the month-to-date invoicing-related metrics. description: The date for the month-to-date invoicing-related metrics.
tests: tests:
- not_null - not_null
- unique
- name: dimension
data_type: string
description: The dimension or granularity of the metrics.
tests:
- accepted_values:
values:
- global
- by_number_of_listings
- name: dimension_value
data_type: string
description: The value or segment available for the selected dimension.
tests:
- not_null