counterweight/data-dwh-dbt-project
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merged PR 3482: Adapts date dimension skeleton for Main KPIs

Browse source 
# Description

New model:
* int_kpis__agg_dates_main_kpis - Serves as the skeleton of dates and dimensions for Main KPIs. It's aggregated since it follows a similar aggregation strategy. It's a single model to feed both Main KPIs visualisations. Note boolean fields are real booleans (true/false) while before these were integers (1/0). This also affects downstream models.

Main KPIs flow adaptations to new skeleton model:
* int_monthly_aggregated_metrics_history_by_deal
* int_monthly_churn_metrics - additionally, calls usual KPIs macro instead of old one
* int_mtd_vs_previous_year_metrics

Reporting changes to ensure report is not down:
* mtd_aggregated_metrics - adaptations on booleans (true-1, false-0)

Cleaning:
* get_kpi_dimensions macro is no longer used
* int_dates_by_deal model and schema entry
* int_dates_mtd_by_dimension model and schema entry
* int_dates_mtd model and schema entry

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #23763
This commit is contained in:
Oriol Roqué Paniagua 2024-11-11 15:57:37 +00:00
parent 2f80642f6c
commit 9ba0edb82d
12 changed files with 197 additions and 416 deletions
Download patch file Download diff file Expand all files Collapse all files

31
macros/business_kpis_configuration.sql
View file

@ -1,33 +1,8 @@
/*
Macro: get_kpi_dimensions
Provides a general configuration for the Dimensions available for the KPIs.
Please note that strings should be encoded with " ' your_value_here ' ",
while fields from tables should be specified like " your_field_here "
*/
{% macro get_kpi_dimensions() %}
{% set dimensions = [
{"dimension": "'global'", "dimension_value": "'global'"},
{
"dimension": "'by_number_of_listings'",
"dimension_value": "active_accommodations_per_deal_segmentation",
},
{
"dimension": "'by_billing_country'",
"dimension_value": "main_billing_country_iso_3_per_deal",
},
] %}
{{ return(dimensions) }}
{% endmacro %}
/*
Macro: get_kpi_dimensions_for_production
Provides the list of Dimensions that will be available in production for the KPIs.
This configuration ensures that working with new dimensions won't affect the display
until all development work has been done.
Additionally, it provides a proper display name for reporting purposes.
Provides the list of Dimensions that will be available in production for the Main KPIs.
It provides a proper display name for reporting purposes.
*/
{% macro get_kpi_dimensions_for_production() %}
@ -90,7 +65,7 @@ Please note that strings should be encoded with " ' your_value_here ' ",
/*
Macro: get_kpi_dimensions_per_model
Provides a general assignemnt for the Dimensions available for each KPI
Provides a general assignement for the Dimensions available for each KPI
model. Keep in mind that these assignations need to be previously
declared.

29
models/intermediate/cross/int_dates_by_deal.sql
View file

@ -1,29 +0,0 @@
{{ config(materialized="table", unique_key=["date", "id_deal"]) }}
with
int_dates as (select * from {{ ref("int_dates") }}),
int_core__unified_user as (select * from {{ ref("int_core__unified_user") }}),
int_core__deal as (select * from {{ ref("int_core__deal") }})
select distinct
d.year_number as year,
d.month_of_year as month,
d.day_of_month as day,
d.date_day as date,
u.id_deal,
deal.main_deal_name,
deal.main_billing_country_iso_3_per_deal,
d.month_start_date as first_day_month,
d.month_end_date as last_day_month
from int_core__unified_user u
inner join int_dates d on d.date_day >= date(u.created_date_utc)
inner join int_core__deal deal on deal.id_deal = u.id_deal
where
-- include only up-to yesterday
now()::date > d.date_day
and (
-- keep all last day of the month
d.date_day = d.month_end_date
-- keep yesterday
or now()::date = d.date_day + 1
)

49
models/intermediate/cross/int_dates_mtd.sql
View file

@ -1,49 +0,0 @@
{{ config(materialized="table", unique_key="date") }}
with
int_dates as (select * from {{ ref("int_dates") }} where date_day >= {{ var("start_date") }}),
raw_dates as (
select
id.year_number as year,
id.month_of_year as month,
id.day_of_month as day,
id.date_day as date,
id.month_start_date as first_day_month,
id.month_end_date as last_day_month,
now()::date as today
from int_dates id
)
-- Dates keeps only end of month (EOM) dates, as well as
-- dates corresponding to the current month and the same month last year
select distinct
rd.year,
rd.month,
rd.day,
rd.date,
rd.first_day_month,
rd.last_day_month,
case when rd.date = rd.last_day_month then 1 else 0 end as is_end_of_month,
case
when date_trunc('month', rd.date) = date_trunc('month', rd.today) then 1
-- If today is 1st of Month, include last day of month for MTD display
when rd.today = rd.last_day_month + 1 then 1
else 0
end as is_current_month
from raw_dates rd
where
rd.today > rd.date
-- include only up-to yesterday
and (
rd.date = rd.last_day_month
-- keep all last day of the month
or rd.date >= date_trunc('month', rd.today)
-- keep all individual days of the current month
or (
rd.year = extract(year from rd.today) - 1
and rd.month = extract(month from rd.today)
and rd.day < extract(day from rd.today)
)
-- keep all days of same month last day up to yesterday's day
)

36
models/intermediate/cross/int_dates_mtd_by_dimension.sql
View file

@ -1,36 +0,0 @@
{% set dimensions = get_kpi_dimensions() %}
{{ config(materialized="table", unique_key=["date", "dimension", "dimension_value"]) }}
with
int_kpis__dimension_daily_accommodation as (
select * from {{ ref("int_kpis__dimension_daily_accommodation") }}
),
int_core__user_host as (select * from {{ ref("int_core__user_host") }}),
int_dates_mtd as (select * from {{ ref("int_dates_mtd") }})
{% for dimension in dimensions %}
select distinct
d.year,
d.month,
d.day,
d.date,
{{ dimension.dimension }} as dimension,
{{ dimension.dimension_value }} as dimension_value,
d.first_day_month,
d.last_day_month,
d.is_end_of_month,
d.is_current_month
from int_dates_mtd d
{% if dimension.dimension == "'by_number_of_listings'" %}
inner join int_kpis__dimension_daily_accommodation a on d.date = a.date
{% elif dimension.dimension == "'by_billing_country'" %}
inner join
int_core__user_host h
on d.date >= date(h.created_date_utc)
and h.main_billing_country_iso_3_per_deal is not null
{% endif %}
{% if not loop.last %}
union all
{% endif %}
{% endfor %}

47
models/intermediate/cross/int_monthly_aggregated_metrics_history_by_deal.sql
View file

@ -3,7 +3,15 @@ This model aggregates the different metrics by deal for those hosts that have it
*/
with
int_dates_by_deal as (select * from {{ ref("int_dates_by_deal") }}),
int_core__deal as (select * from {{ ref("int_core__deal") }}),
int_kpis__agg_dates_main_kpis as (
select *
from {{ ref("int_kpis__agg_dates_main_kpis") }}
where
dimension in ('by_deal')
and dimension_value <> 'UNSET'
and is_end_of_month = true
),
daily_deal_lifecycle as (select * from {{ ref("int_kpis__lifecycle_daily_deal") }}),
listings as (
select *
@ -74,9 +82,10 @@ select
d.month,
d.day,
d.date,
d.id_deal,
d.main_deal_name,
d.main_billing_country_iso_3_per_deal,
-- DEAL STATIC ATTRIBUTES --
icd.id_deal,
icd.main_deal_name,
icd.main_billing_country_iso_3_per_deal,
-- DEAL LIFECYCLE --
daily_deal_lifecycle.deal_lifecycle_state,
@ -179,53 +188,55 @@ select
listings.listings_booked_in_month, 0
) as total_revenue_per_listings_booked_in_month
from int_dates_by_deal d
from int_kpis__agg_dates_main_kpis d
left join int_core__deal icd on d.dimension_value = icd.id_deal
left join
daily_deal_lifecycle
on d.date = daily_deal_lifecycle.date
and d.id_deal = daily_deal_lifecycle.id_deal
and d.dimension_value = daily_deal_lifecycle.id_deal
left join
created_bookings
on d.date = created_bookings.end_date
and d.id_deal = created_bookings.dimension_value
and d.dimension_value = created_bookings.dimension_value
left join
check_out_bookings
on d.date = check_out_bookings.end_date
and d.id_deal = check_out_bookings.dimension_value
and d.dimension_value = check_out_bookings.dimension_value
left join
cancelled_bookings
on d.date = cancelled_bookings.end_date
and d.id_deal = cancelled_bookings.dimension_value
and d.dimension_value = cancelled_bookings.dimension_value
left join
billable_bookings
on d.date = billable_bookings.end_date
and d.id_deal = billable_bookings.dimension_value
and d.dimension_value = billable_bookings.dimension_value
left join
created_guest_journeys
on d.date = created_guest_journeys.end_date
and d.id_deal = created_guest_journeys.dimension_value
and d.dimension_value = created_guest_journeys.dimension_value
left join
started_guest_journeys
on d.date = started_guest_journeys.end_date
and d.id_deal = started_guest_journeys.dimension_value
and d.dimension_value = started_guest_journeys.dimension_value
left join
completed_guest_journeys
on d.date = completed_guest_journeys.end_date
and d.id_deal = completed_guest_journeys.dimension_value
and d.dimension_value = completed_guest_journeys.dimension_value
left join
guest_journeys_with_payment
on d.date = guest_journeys_with_payment.end_date
and d.id_deal = guest_journeys_with_payment.dimension_value
and d.dimension_value = guest_journeys_with_payment.dimension_value
left join
guest_payments
on d.date = guest_payments.end_date
and d.id_deal = guest_payments.dimension_value
left join listings on d.date = listings.date and d.id_deal = listings.dimension_value
and d.dimension_value = guest_payments.dimension_value
left join
listings on d.date = listings.date and d.dimension_value = listings.dimension_value
left join
invoiced_revenue
on d.date = invoiced_revenue.end_date
and d.id_deal = invoiced_revenue.dimension_value
and d.dimension_value = invoiced_revenue.dimension_value
left join
host_resolutions
on d.date = host_resolutions.end_date
and d.id_deal = host_resolutions.dimension_value
and d.dimension_value = host_resolutions.dimension_value

14
models/intermediate/cross/int_monthly_churn_metrics.sql
View file

@ -1,4 +1,4 @@
{% set dimensions = get_kpi_dimensions() %}
{% set dimensions = get_kpi_dimensions_per_model("CHURN_RATES") %}
{% set churn_lifecycle_states = "('05-Churning')" %}
{{ config(materialized="table", unique_key=["date", "dimension", "dimension_value"]) }}
@ -6,8 +6,12 @@ with
int_monthly_12m_window_contribution_by_deal as (
select * from {{ ref("int_monthly_12m_window_contribution_by_deal") }}
),
int_dates_mtd_by_dimension as (
select * from {{ ref("int_dates_mtd_by_dimension") }}
int_kpis__agg_dates_main_kpis as (
select *
from {{ ref("int_kpis__agg_dates_main_kpis") }}
where
dimension in ('global', 'by_number_of_listings', 'by_billing_country')
and dimension_value <> 'UNSET'
),
int_kpis__dimension_daily_accommodation as (
select * from {{ ref("int_kpis__dimension_daily_accommodation") }}
@ -78,7 +82,7 @@ select
cast(
c.listings_booked_in_month_churn_average_contribution as numeric(19, 6)
) as listings_booked_in_month_churn_average_contribution
from int_dates_mtd_by_dimension d
from int_kpis__agg_dates_main_kpis d
left join
churn_metrics_per_date c
on c.date = d.date
@ -86,4 +90,4 @@ left join
and c.dimension_value = d.dimension_value
-- Remove current month dates since data won't be available anyway. This is specific
-- for this churn metrics model
where d.is_current_month = 0
where d.is_current_month = false

14
models/intermediate/cross/int_mtd_vs_previous_year_metrics.sql
View file

@ -164,8 +164,12 @@ with
and dimension_value <> 'UNSET'
),
int_monthly_churn_metrics as (select * from {{ ref("int_monthly_churn_metrics") }}),
int_dates_mtd_by_dimension as (
select * from {{ ref("int_dates_mtd_by_dimension") }}
int_kpis__agg_dates_main_kpis as (
select *
from {{ ref("int_kpis__agg_dates_main_kpis") }}
where
dimension in ('global', 'by_number_of_listings', 'by_billing_country')
and dimension_value <> 'UNSET'
),
plain_kpi_combination as (
@ -308,7 +312,7 @@ with
churn.created_bookings_churn_average_contribution,
churn.listings_booked_in_month_churn_average_contribution
from int_dates_mtd_by_dimension d
from int_kpis__agg_dates_main_kpis d
left join
created_bookings
on d.date = created_bookings.end_date
@ -498,6 +502,6 @@ left join
and current.year = previous_year.year + 1
where
(
current.is_end_of_month = 1
or (current.is_current_month = 1 and current.day = previous_year.day)
current.is_end_of_month = true
or (current.is_current_month = true and current.day = previous_year.day)
)

243
models/intermediate/cross/schema.yml
View file

@ -175,151 +175,6 @@ models:
tests:
- not_null
- name: int_dates_mtd
description: |
This model provides Month-To-Date (MTD) necessary dates for MTD-based models to work.
- For month-to-month complete information, it retrieves all end month dates that have elapsed since 2020.
- For month-to-date information, it retrieves the days of the current month of this year up to yesterday.
Additionally, it also gets the days of its equivalent month from last year previous the current day of month of today.
Example:
Imagine we have are at 4th June 2024.
- We will get the dates for 1st, 2nd, 3rd of June 2024.
- We will also get the dates for 1st, 2nd, 3rd of June 2023.
- We will get all end of months from 2020 to yesterday,
i.e., 31st January 2020, 29th February 2020, ..., 30th April 2024, 31st May 2024.
deprecation_date: 2024-11-30
columns:
- name: year
data_type: int
description: Year number of the given date.
tests:
- not_null
- name: month
data_type: int
description: Month number of the given date.
tests:
- not_null
- name: day
data_type: int
description: Day monthly number of the given date.
tests:
- not_null
- name: is_end_of_month
data_type: boolean
description: Is end of month, 1 for yes, 0 for no.
tests:
- not_null
- name: is_current_month
data_type: boolean
description: |
Checks if the date is within the current executed month,
1 for yes, 0 for no.
tests:
- not_null
- name: first_day_month
data_type: date
description: |
First day of the month correspoding to the date field.
It comes from int_dates_mtd logic.
tests:
- not_null
- name: date
data_type: date
description: |
Main date for the computation, that is used for filters.
It's the primary key for this model.
tests:
- not_null
- unique
- name: int_dates_by_deal
description: |
This model provides the necessary dates for each deal for deal-based KPIs models to work.
It only considers those dates starting from when the host user of the deal was first available.
deprecation_date: 2024-11-30
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- date
- id_deal
columns:
- name: year
data_type: int
description: Year number of the given date.
tests:
- not_null
- name: month
data_type: int
description: Month number of the given date.
tests:
- not_null
- name: day
data_type: int
description: Day monthly number of the given date.
tests:
- not_null
- name: last_day_month
data_type: date
description: |
Last day of the month correspoding to the date field.
It comes from int_dates_mtd logic.
tests:
- not_null
- name: first_day_month
data_type: date
description: |
First day of the month correspoding to the date field.
It comes from int_dates_mtd logic.
tests:
- not_null
- name: date
data_type: date
description: |
Main date for the computation, that is used for filters.
It's the primary key for this model.
tests:
- not_null
- name: id_deal
data_type: string
description: |
Main identifier of the B2B clients. A deal can have multiple hosts.
A host should usually have a deal, but it does not happen on all cases.
In this KPI reporting we force that Deal is not null to avoid potential
data quality issues.
tests:
- not_null
- name: main_deal_name
data_type: string
description: |
Main name for this ID deal.
tests:
- not_null
- name: main_billing_country_iso_3_per_deal
data_type: string
description: |
ISO 3166-1 alpha-3 main country code in which the Deal is billed.
In some cases it's null.
- name: int_mtd_aggregated_metrics
description: |
The `int_mtd_aggregated_metrics` model aggregates multiple metrics on a year, month, and day basis.
@ -372,16 +227,14 @@ models:
- name: first_day_month
data_type: date
description: |
first day of the month correspoding to the date field.
It comes from int_dates_mtd logic.
first day of the month corresponding to the date field.
tests:
- not_null
- name: date
data_type: date
description: |
main date for the computation, that is used for filters.
It comes from int_dates_mtd logic.
main date for the computation, that is used for filters.
tests:
- not_null
@ -405,7 +258,7 @@ models:
data_type: date
description: |
corresponds to the date of the previous year, with respect to the field date.
It comes from int_dates_mtd logic. It's only displayed for information purposes,
It's only displayed for information purposes,
should not be needed for reporting.
- name: metric
@ -506,96 +359,6 @@ models:
ISO 3166-1 alpha-3 main country code in which the Deal is billed.
In some cases it's null.
- name: int_dates_mtd_by_dimension
description: |
This model provides Month-To-Date (MTD) necessary dates, dimension and dimension_values
for MTD-based models to work.
It provides the basic "empty" structure from which metrics will be built upon. This is, on
top of the Date that characterises int_dates_mtd, including the dimensions and their
respective values that should appear in any mtd metric model.
Example:
- For the "global" dimension, we will only have the "global" dimension value.
- For the "by_number_of_listing" dimension, we will have different values
according to the segments defined, ex: 0, 1-5, 6-20, etc.
... and so on and forth for any available dimension. These combinations should appear
for each date of the MTD models.
deprecation_date: 2024-11-30
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- date
- dimension
- dimension_value
columns:
- name: year
data_type: int
description: Year number of the given date.
tests:
- not_null
- name: month
data_type: int
description: Month number of the given date.
tests:
- not_null
- name: day
data_type: int
description: Day monthly number of the given date.
tests:
- not_null
- name: is_end_of_month
data_type: boolean
description: Is end of month, 1 for yes, 0 for no.
tests:
- not_null
- name: is_current_month
data_type: boolean
description: |
Checks if the date is within the current executed month,
1 for yes, 0 for no.
tests:
- not_null
- name: first_day_month
data_type: date
description: |
First day of the month correspoding to the date field.
It comes from int_dates_mtd logic.
tests:
- not_null
- name: date
data_type: date
description: |
Main date for the computation, metrics include monthly information
until this date.
tests:
- not_null
- name: dimension
data_type: string
description: The dimension or granularity of the metrics.
tests:
- accepted_values:
values:
- global
- by_number_of_listings
- by_billing_country
- name: dimension_value
data_type: string
description: The value or segment available for the selected dimension.
tests:
- not_null
- name: int_monthly_growth_score_by_deal
description: |
The main goal of this model is to provide a growth score by deal and month.

53
models/intermediate/kpis/int_kpis__agg_dates_main_kpis.sql Normal file
View file

@ -0,0 +1,53 @@
{% set dimensions = get_kpi_dimensions_per_model("") %}
{{ config(materialized="table", unique_key=["date", "dimension", "dimension_value"]) }}
with
daily_dim as (
select distinct
ikdd.year,
ikdd.month,
ikdd.day,
ikdd.date,
-- Dimensions --
coalesce(icuh.id_deal, 'UNSET') as id_deal,
coalesce(
icd.main_billing_country_iso_3_per_deal, 'UNSET'
) as main_billing_country_iso_3_per_deal,
coalesce(
icmas.active_accommodations_per_deal_segmentation, 'UNSET'
) as active_accommodations_per_deal_segmentation,
ikdd.first_day_month,
ikdd.last_day_month,
ikdd.is_end_of_month,
ikdd.is_current_month
from {{ ref("int_kpis__dimension_dates") }} as ikdd
left join
{{ ref("int_core__user_host") }} as icuh
on ikdd.date >= date(icuh.created_date_utc)
left join {{ ref("int_core__deal") }} as icd on icuh.id_deal = icd.id_deal
left join
{{ ref("int_kpis__dimension_daily_accommodation") }} as icmas
on icuh.id_deal = icmas.id_deal
and ikdd.date = icmas.date
where (ikdd.is_month_to_date = true or ikdd.is_end_of_month)
)
{% for dimension in dimensions %}
select distinct
year,
month,
day,
date,
{{ dimension.dimension }} as dimension,
{{ dimension.dimension_value }} as dimension_value,
first_day_month,
last_day_month,
is_end_of_month,
is_current_month
from daily_dim
where {{ dimension.dimension_value }} <> 'UNSET'
{% if not loop.last %}
union all
{% endif %}
{% endfor %}

89
models/intermediate/kpis/schema.yml
View file

@ -41,14 +41,14 @@ models:
- name: first_day_month
data_type: date
description: |
First day of the month correspoding to the date field.
First day of the month corresponding to the date field.
tests:
- not_null
- name: last_day_month
data_type: date
description: |
Last day of the month correspoding to the date field.
Last day of the month corresponding to the date field.
tests:
- not_null
@ -103,6 +103,91 @@ models:
tests:
- not_null
- name: int_kpis__agg_dates_main_kpis
description: |
This model provides the skeleton of dates and dimensions needed for Main KPIs display.
It encapsulates the multiple manners to present data in the reporting, namely, Monthly+MTD
per a given dimension or specifically Monthly by Deal.
The rest of the metrics computed are attached to this master table.
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- date
- dimension
- dimension_value
columns:
- name: date
data_type: date
description: |
The end date of the time range considered for the metrics that will be
available in this record.
tests:
- not_null
- name: dimension
data_type: string
description: The dimension or granularity of the metrics.
tests:
- accepted_values:
values:
- global
- by_number_of_listings
- by_billing_country
- by_deal
- name: dimension_value
data_type: string
description: The value or segment available for the selected dimension.
tests:
- not_null
- name: year
data_type: int
description: Year number of the given date.
tests:
- not_null
- name: month
data_type: int
description: Month number of the given date.
tests:
- not_null
- name: day
data_type: int
description: Day monthly number of the given date.
tests:
- not_null
- name: first_day_month
data_type: date
description: |
First day of the month corresponding to the date field.
tests:
- not_null
- name: last_day_month
data_type: date
description: |
Last day of the month corresponding to the date field.
tests:
- not_null
- name: is_end_of_month
data_type: boolean
description: True if it's end of month, false otherwise.
tests:
- not_null
- name: is_current_month
data_type: boolean
description: |
True if the date is within the current month, false otherwise.
tests:
- not_null
- name: int_kpis__lifecycle_daily_accommodation
description: |
This model computes the daily lifecycle segment for each accommodation, also known as

6
models/reporting/general/mtd_aggregated_metrics.sql
View file

@ -24,8 +24,8 @@ select
year as year,
month as month,
day as day,
is_end_of_month as is_end_of_month,
is_current_month as is_current_month,
case when is_end_of_month then 1 else 0 end as is_end_of_month,
case when is_current_month then 1 else 0 end as is_current_month,
first_day_month as first_day_month,
date as date,
dimension_display as dimension,
@ -80,4 +80,4 @@ where
)
)
-- If metric is Churn Rate, do not show month in progress
and not (lower(metric) like '%churn rate%' and is_current_month = 1)
and not (lower(metric) like '%churn rate%' and is_current_month = true)

2
models/reporting/general/schema.yml
View file

@ -358,7 +358,7 @@ models:
- name: first_day_month
data_type: date
description: |
First day of the month correspoding to the date field.
First day of the month corresponding to the date field.
It comes from int_dates_mtd logic.
tests:
- not_null