counterweight/data-dwh-dbt-project
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merged PR 2689: KPIs by Billing Country

Browse source 
# Description

Adds Billing Country dimension in KPIs, but does not expose them to reporting yet.
Silly thing, based on the macros I built, I cannot make incremental changes unless changing all models. This will need to be adapted, happy to hear your thoughts on how we do it.
Additionally, I have lack of performance of the model `mtd_guest_payments_metrics`. It takes around 5 min to execute, but technically the end-to-end runs in one shoot without breaking.
It's a complex PR because it changes many files, but you will see that:
* It mostly changes the join conditions for the dimensions or the schema tests,
* I tried to be very careful and add things step-by-step in the commits.

Goal is NOT to complete the PR yet until we see how we can improve performance. I can say though that data end-to-end looks ok to me, but would benefit from checking with production data for the new dimension

Update 30th Aug
* Added a new commit that includes `id_user_host` in `int_core__verification_payments`. Happy to discuss if it makes sense or not. But it changes the execution from ~600 sec to ~6 sec because it avoids a massive repeated join with `verification_requests`.

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models. **To check because of performance issues**

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #19082
This commit is contained in:
Oriol Roqué Paniagua 2024-09-04 10:17:12 +00:00
parent 940896824f
commit 556a52e991
12 changed files with 95 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

5
models/intermediate/core/int_core__mtd_accommodation_metrics.sql
View file

@ -96,6 +96,11 @@ with
int_core__mtd_accommodation_segmentation mas
on u.id_deal = mas.id_deal
and al.date = mas.date
{% elif dimension.dimension == "'by_billing_country'" %}
inner join int_core__unique_accommodation_to_user atu
on atu.id_accommodation = al.id_accommodation
inner join int_core__user_host u on atu.id_user_owner = u.id_user_host
and u.billing_country_iso_3 is not null
{% endif %}
group by 1, 2, 3
{% if not loop.last %}

13
models/intermediate/core/int_core__mtd_booking_metrics.sql
View file

@ -36,6 +36,9 @@ with
inner join int_core__mtd_accommodation_segmentation mas
on u.id_deal = mas.id_deal
and d.date = mas.date
{% elif dimension.dimension == "'by_billing_country'" %}
inner join int_core__user_host u on b.id_user_host = u.id_user_host
and u.billing_country_iso_3 is not null
{% endif %}
group by 1, 2, 3
{% if not loop.last %}
@ -60,6 +63,9 @@ with
inner join int_core__mtd_accommodation_segmentation mas
on u.id_deal = mas.id_deal
and d.date = mas.date
{% elif dimension.dimension == "'by_billing_country'" %}
inner join int_core__user_host u on b.id_user_host = u.id_user_host
and u.billing_country_iso_3 is not null
{% endif %}
group by 1, 2, 3
{% if not loop.last %}
@ -85,6 +91,9 @@ with
inner join int_core__mtd_accommodation_segmentation mas
on u.id_deal = mas.id_deal
and d.date = mas.date
{% elif dimension.dimension == "'by_billing_country'" %}
inner join int_core__user_host u on b.id_user_host = u.id_user_host
and u.billing_country_iso_3 is not null
{% endif %}
group by 1, 2, 3
{% if not loop.last %}
@ -111,6 +120,10 @@ with
int_core__mtd_accommodation_segmentation mas
on u.id_deal = mas.id_deal
and d.date = mas.date
{% elif dimension.dimension == "'by_billing_country'" %}
inner join int_core__bookings b on b.id_booking = bce.id_booking
inner join int_core__user_host u on b.id_user_host = u.id_user_host
and u.billing_country_iso_3 is not null
{% endif %}
group by 1, 2, 3
{% if not loop.last %}

7
models/intermediate/core/int_core__mtd_deal_metrics.sql
View file

@ -17,7 +17,9 @@ with
int_core__mtd_accommodation_segmentation as (
select * from {{ ref("int_core__mtd_accommodation_segmentation") }}
),
int_core__user_host as (
select * from {{ ref("int_core__user_host") }}
),
deals_metric_aggregation_per_date as (
{% for dimension in dimensions %}
select
@ -88,6 +90,9 @@ with
int_core__mtd_accommodation_segmentation mas
on al.id_deal = mas.id_deal
and al.date = mas.date
{% elif dimension.dimension == "'by_billing_country'" %}
inner join int_core__user_host u on al.id_deal = u.id_deal
and u.billing_country_iso_3 is not null
{% endif %}
group by 1, 2, 3
{% if not loop.last %}

17
models/intermediate/core/int_core__mtd_guest_journey_metrics.sql
View file

@ -49,6 +49,10 @@ with
inner join int_core__mtd_accommodation_segmentation mas
on u.id_deal = mas.id_deal
and d.date = mas.date
{% elif dimension.dimension == "'by_billing_country'" %}
inner join int_core__user_host u
on vr.id_user_host = u.id_user_host
and u.billing_country_iso_3 is not null
{% endif %}
group by 1, 2, 3
{% if not loop.last %}
@ -74,6 +78,10 @@ with
inner join int_core__mtd_accommodation_segmentation mas
on u.id_deal = mas.id_deal
and d.date = mas.date
{% elif dimension.dimension == "'by_billing_country'" %}
inner join int_core__user_host u
on vr.id_user_host = u.id_user_host
and u.billing_country_iso_3 is not null
{% endif %}
group by 1, 2, 3
{% if not loop.last %}
@ -99,6 +107,10 @@ with
inner join int_core__mtd_accommodation_segmentation mas
on u.id_deal = mas.id_deal
and d.date = mas.date
{% elif dimension.dimension == "'by_billing_country'" %}
inner join int_core__user_host u
on vr.id_user_host = u.id_user_host
and u.billing_country_iso_3 is not null
{% endif %}
group by 1, 2, 3
{% if not loop.last %}
@ -125,6 +137,11 @@ with
inner join int_core__mtd_accommodation_segmentation mas
on u.id_deal = mas.id_deal
and d.date = mas.date
{% elif dimension.dimension == "'by_billing_country'" %}
inner join int_core__verification_requests vr on vr.id_verification_request = p.id_verification_request
inner join int_core__user_host u
on vr.id_user_host = u.id_user_host
and u.billing_country_iso_3 is not null
{% endif %}
group by 1, 2, 3
{% if not loop.last %}

8
models/intermediate/core/int_core__mtd_guest_payments_metrics.sql
View file

@ -65,14 +65,14 @@ with
on date_trunc('month', vp.payment_paid_date_utc)::date = d.first_day_month
and extract(day from vp.payment_paid_date_utc) <= d.day
{% if dimension.dimension == "'by_number_of_listings'" %}
inner join
int_core__verification_requests vr
on vp.id_verification_request = vr.id_verification_request
inner join int_core__user_host u on vr.id_user_host = u.id_user_host
inner join int_core__user_host u on vp.id_user_host = u.id_user_host
inner join
int_core__mtd_accommodation_segmentation mas
on u.id_deal = mas.id_deal
and d.date = mas.date
{% elif dimension.dimension == "'by_billing_country'" %}
inner join int_core__user_host u on vp.id_user_host = u.id_user_host
and u.billing_country_iso_3 is not null
{% endif %}
where upper(vp.payment_status) = {{ var("paid_payment_state") }}
group by 1, 2, 3

8
models/intermediate/core/int_core__verification_payments.sql
View file

@ -7,6 +7,7 @@ with
select * from {{ ref("stg_core__verification_payment_type") }}
),
stg_core__verification as (select * from {{ ref("stg_core__verification") }}),
stg_core__verification_request as (select * from {{ ref("stg_core__verification_request") }}),
stg_core__payment as (select * from {{ ref("stg_core__payment") }}),
stg_core__payment_status as (select * from {{ ref("stg_core__payment_status") }}),
int_simple_exchange_rates as (select * from {{ ref("int_simple_exchange_rates") }})
@ -26,6 +27,10 @@ select
p.refunded_at_utc as payment_refunded_at_utc,
p.refunded_date_utc as payment_refunded_date_utc,
p.refund_payment_reference,
-- Host User identifier is included to speed up
-- KPIs execution, even though the host itself
-- has nothing to do with the guest payments.
vr.id_user_host,
vtp.id_guest_user,
vtp.id_verification,
v.id_verification_request,
@ -47,3 +52,6 @@ left join
on vtp.payment_due_date_utc = r.rate_date_utc
and p.currency = r.from_currency
and r.to_currency = 'GBP'
left join
stg_core__verification_request vr
on v.id_verification_request = vr.id_verification_request

12
models/intermediate/core/schema.yaml
View file

@ -542,6 +542,7 @@ models:
values:
- global
- by_number_of_listings
- by_billing_country
- name: dimension_value
data_type: string
@ -577,6 +578,7 @@ models:
values:
- global
- by_number_of_listings
- by_billing_country
- name: dimension_value
data_type: string
@ -612,6 +614,7 @@ models:
values:
- global
- by_number_of_listings
- by_billing_country
- name: dimension_value
data_type: string
@ -647,6 +650,7 @@ models:
values:
- global
- by_number_of_listings
- by_billing_country
- name: dimension_value
data_type: string
@ -682,6 +686,7 @@ models:
values:
- global
- by_number_of_listings
- by_billing_country
- name: dimension_value
data_type: string
@ -870,6 +875,13 @@ models:
data_type: date
- name: refund_payment_reference
data_type: character varying
- name: id_user_host
data_type: character varying
description: |
UUID of the Host linked to the Verification Request
that has a payment. It's included here to speed up
KPIs execution, even though the host itself has nothing
to do with the guest payments.
- name: id_guest_user
data_type: character varying
- name: id_verification

7
models/intermediate/cross/int_dates_mtd_by_dimension.sql
View file

@ -7,6 +7,9 @@ with
int_core__mtd_accommodation_segmentation as (
select * from {{ ref("int_core__mtd_accommodation_segmentation") }}
),
int_core__user_host as (
select * from {{ ref("int_core__user_host") }}
),
int_dates_mtd as (select * from {{ ref("int_dates_mtd") }})
{% for dimension in dimensions %}
@ -25,6 +28,10 @@ with
{% if dimension.dimension == "'by_number_of_listings'" %}
inner join int_core__mtd_accommodation_segmentation a
on d.date = a.date
{% elif dimension.dimension == "'by_billing_country'" %}
inner join int_core__user_host h
on d.date >= h.created_date_utc
and h.billing_country_iso_3 is not null
{% endif %}
{% if not loop.last %}
union all

3
models/intermediate/cross/schema.yml
View file

@ -167,6 +167,7 @@ models:
values:
- global
- by_number_of_listings
- by_billing_country
- name: dimension_value
data_type: string
@ -375,6 +376,7 @@ models:
values:
- global
- by_number_of_listings
- by_billing_country
- name: dimension_value
data_type: string
@ -549,6 +551,7 @@ models:
values:
- global
- by_number_of_listings
- by_billing_country
- name: dimension_value
data_type: string

17
models/intermediate/xero/int_xero__mtd_invoicing_metrics.sql
View file

@ -34,6 +34,7 @@ with
select * from {{ ref("int_dates_mtd_by_dimension") }}
),
int_xero__contacts as (select * from {{ ref("int_xero__contacts") }}),
int_core__user_host as (select * from {{ ref("int_core__user_host") }}),
resolution_host_payment as (
{% for dimension in dimensions %}
select
@ -62,6 +63,12 @@ with
int_core__mtd_accommodation_segmentation mas
on c.id_deal = mas.id_deal
and d.date = mas.date
{% elif dimension.dimension == "'by_billing_country'" %}
inner join int_xero__contacts c on c.id_contact = bt.id_contact
inner join
int_core__user_host u
on c.id_deal = u.id_deal
and u.billing_country_iso_3 is not null
{% endif %}
group by 1, 2, 3
{% if not loop.last %}
@ -99,6 +106,11 @@ with
int_core__mtd_accommodation_segmentation mas
on sdm.id_deal = mas.id_deal
and d.date = mas.date
{% elif dimension.dimension == "'by_billing_country'" %}
inner join
int_core__user_host u
on sdm.id_deal = u.id_deal
and u.billing_country_iso_3 is not null
{% endif %}
where
upper(sdm.document_status) in {{ relevant_document_statuses }}
@ -156,6 +168,11 @@ with
int_core__mtd_accommodation_segmentation mas
on sdm.id_deal = mas.id_deal
and d.date = mas.date
{% elif dimension.dimension == "'by_billing_country'" %}
inner join
int_core__user_host u
on sdm.id_deal = u.id_deal
and u.billing_country_iso_3 is not null
{% endif %}
where
upper(sdm.document_status) in {{ relevant_document_statuses }}

1
models/intermediate/xero/schema.yaml
View file

@ -313,6 +313,7 @@ models:
values:
- global
- by_number_of_listings
- by_billing_country
- name: dimension_value
data_type: string