data-dwh-dbt-project/models/intermediate/cross/schema.yml

1709 lines
60 KiB
YAML
Raw Normal View History

2024-06-14 15:48:24 +02:00
models:
- name: int_daily_currency_exchange_rates
description: >-
This model holds a lot of data on currency exchange rates. The time
granularity is daily. Each record holds a currency pair for a specific
day, source and version.
2024-06-14 16:22:00 +02:00
Actual rates are sourced from xe.com data. The `guessed` and `forecast`
2024-06-14 15:48:24 +02:00
versions are built by simply 'pushing' the first/last exchange rate on
record. Basically, wherever we dont' have data for a date, we pick the
2024-06-14 16:22:00 +02:00
closest actual data point that comes from xe.com. Bear in mind this means
that `forecast` version records will change on a daily basis as actual
data moves forwards, meaning you shouldn't assume your money amounts
converted in the future should always stay put.
2024-06-14 15:48:24 +02:00
Note that, given the dimensionality, getting a simple time series for a
currency pair will require a bit of filtering.
Reverse rates are explicit. This means that, for any given day and any
given currency pair, you will find two records with opposite from/to
positions. So, for 2024-01-01, you will find both a EUR->USD record and a
USD->EUR record with the opposite rate (1/rate).
columns:
- name: id_exchange_rate
data_type: text
description: A unique ID for the record, derived from concatenating the
currencies, date, source and version. Currency order is relevant
(EURUSD != USDEUR).
tests:
- not_null
- unique
- name: from_currency
data_type: character
description: The source currency, represented as an ISO 4217 code.
tests:
- not_null
- name: to_currency
data_type: character
description: The target currency, represented as an ISO 4217 code.
tests:
- not_null
- name: rate
data_type: numeric
description: >-
The exchange rate, represented as the units of the target currency
that one unit of source currency gets you. So, from_currency=USD to
to_currency=PLN with rate=4.2 should be read as '1 US Dollar buys me
4.2 Polish Zlotys'.
For same currency pairs (EUR to EUR, USD to USD, etc). The rate will
always be one.
The rate can be smaller than one, but can't be negative.
tests:
- not_negative_or_zero
- not_null
- name: rate_date_utc
data_type: date
description: The date in which the rate record is relevant.
tests:
- not_null
- name: source
data_type: text
2024-06-14 16:46:28 +02:00
description:
Where is the data coming from. Records that are composed from
2024-06-14 15:48:24 +02:00
making assumptions on real data will contain `_inferred`.
- name: rate_version
data_type: text
2024-06-14 16:46:28 +02:00
description:
The version of the rate. This can be one of `actual` (the rate is a
2024-06-14 15:48:24 +02:00
reality fact), `forecast` (the rate sits in the future and is a guess
in nature) or `guess` (the rate sits in the past and is a guess in
nature). Note that one currency pair can have multiple rate versions
on the same date.
tests:
- accepted_values:
values:
- guess
- actual
- forecast
2024-06-14 16:22:00 +02:00
- not_null
2024-06-14 15:48:24 +02:00
- name: updated_at_utc
data_type: timestamp with time zone
2024-06-14 16:46:28 +02:00
description:
For external sources, this will be the point in time when the
2024-06-14 15:48:24 +02:00
information was obtained from them. For stuff we make up here in the
DWH, this will be the point in time when we made the assumption.
tests:
- not_null
2024-06-14 16:44:48 +02:00
- name: int_simple_exchange_rates
2024-06-14 16:46:28 +02:00
description: >-
A simplified vision of exchange rates, derived from
`int_daily_currency_exchange_rates`. Come here if you don't want to
understand nuances and complexities and just want to convert rates.
The time granularity is daily. Each record holds a currency pair for a
specific day. You will only find one conversion rate per currency pair and
date.
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- from_currency
- to_currency
- rate_date_utc
2024-06-14 16:44:48 +02:00
columns:
- name: from_currency
data_type: character
description: The source currency, represented as an ISO 4217 code.
tests:
- not_null
- name: to_currency
data_type: character
description: The source currency, represented as an ISO 4217 code.
tests:
- not_null
- name: rate
data_type: numeric
description: The target currency, represented as an ISO 4217 code.
tests:
- not_null
- name: rate_date_utc
data_type: date
description: The date in which the rate record is relevant.
tests:
- not_null
- name: updated_at_utc
data_type: timestamp with time zone
2024-06-14 16:46:28 +02:00
description:
For external sources, this will be the point in time when the
2024-06-14 16:44:48 +02:00
information was obtained from them. For stuff we make up here in the
DWH, this will be the point in time when we made the assumption.
tests:
- not_null
- name: int_mtd_vs_previous_year_metrics
description: |
This model is used for global KPIs.
It aggregates all the mtd models with the different metrics per source
and computes any necessary weighted metric across different sources.
Merged PR 2607: Propagates and exposes multiple dimension handling for KPIs # Description This PR ensures the propagation of the dimensions for KPIs across the key aggregating and exposing models. Additionally, provides these 2 new fields in reporting while **not affecting the current data display**, thus it's safe to work in the PBI report without needing to work in 2 PRs in parallel. **Changes:** **1 - Intermediate, `int_mtd_vs_previous_year_metrics`:** * Removes the temporary filter on `where dimension in ({{ production_dimensions }})`. This will be applied directly to reporting later. This ensures that the new dimension on customer segmentation is fully available only within intermediate. * Adds `dimension` and `dimension_value` granularity. This includes: 1) adding these fields, 2) joining by these fields with all the source CTEs containing the source models with metrics - which in turn needs the change of the dates model - and 3) joining by these fields in the self-join to compute the incremental vs. previous year. * Changes on the schema file **2 - Intermediate, `int_mtd_aggregated_metrics`:** * Adds `dimension` and `dimension_value` granularity. This includes only adding these fields. * Changes on the schema file **3 - Reporting, `mtd_aggregated_metrics`:** * Adds the filter removed on `int_mtd_vs_previous_year_metrics`. This ensures that only the Global dimension is available for the reporting, thus **no changes from user POV**. * Adds `dimension` and `dimension_value` granularity. This includes only adding these fields * Changes on the schema file # Checklist - [X] The edited models and dependants run properly with production data. - [X] The edited models are sufficiently documented. - [X] The edited models contain PK tests, and I've ran and passed them. - [X] I have checked for DRY opportunities with other models and docs. - [X] I've picked the right materialization for the affected models. # Other - [ ] Check if a full-refresh is required after this PR is merged. Related work items: #19325
2024-08-20 15:42:27 +00:00
Each metric has a date, dimension and dimension value that defines
the primary key of this model.
2024-09-12 15:38:50 +02:00
Finally, it displays any metric on the current date, the previous year
date and it computes the relative increment by using the macro:
- calculate_safe_relative_increment
2024-09-12 15:38:50 +02:00
tests:
Merged PR 2607: Propagates and exposes multiple dimension handling for KPIs # Description This PR ensures the propagation of the dimensions for KPIs across the key aggregating and exposing models. Additionally, provides these 2 new fields in reporting while **not affecting the current data display**, thus it's safe to work in the PBI report without needing to work in 2 PRs in parallel. **Changes:** **1 - Intermediate, `int_mtd_vs_previous_year_metrics`:** * Removes the temporary filter on `where dimension in ({{ production_dimensions }})`. This will be applied directly to reporting later. This ensures that the new dimension on customer segmentation is fully available only within intermediate. * Adds `dimension` and `dimension_value` granularity. This includes: 1) adding these fields, 2) joining by these fields with all the source CTEs containing the source models with metrics - which in turn needs the change of the dates model - and 3) joining by these fields in the self-join to compute the incremental vs. previous year. * Changes on the schema file **2 - Intermediate, `int_mtd_aggregated_metrics`:** * Adds `dimension` and `dimension_value` granularity. This includes only adding these fields. * Changes on the schema file **3 - Reporting, `mtd_aggregated_metrics`:** * Adds the filter removed on `int_mtd_vs_previous_year_metrics`. This ensures that only the Global dimension is available for the reporting, thus **no changes from user POV**. * Adds `dimension` and `dimension_value` granularity. This includes only adding these fields * Changes on the schema file # Checklist - [X] The edited models and dependants run properly with production data. - [X] The edited models are sufficiently documented. - [X] The edited models contain PK tests, and I've ran and passed them. - [X] I have checked for DRY opportunities with other models and docs. - [X] I've picked the right materialization for the affected models. # Other - [ ] Check if a full-refresh is required after this PR is merged. Related work items: #19325
2024-08-20 15:42:27 +00:00
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- date
- dimension
- dimension_value
2024-09-12 15:38:50 +02:00
columns:
- name: date
data_type: date
description: The date for the month-to-date metrics.
tests:
- not_null
Merged PR 2607: Propagates and exposes multiple dimension handling for KPIs # Description This PR ensures the propagation of the dimensions for KPIs across the key aggregating and exposing models. Additionally, provides these 2 new fields in reporting while **not affecting the current data display**, thus it's safe to work in the PBI report without needing to work in 2 PRs in parallel. **Changes:** **1 - Intermediate, `int_mtd_vs_previous_year_metrics`:** * Removes the temporary filter on `where dimension in ({{ production_dimensions }})`. This will be applied directly to reporting later. This ensures that the new dimension on customer segmentation is fully available only within intermediate. * Adds `dimension` and `dimension_value` granularity. This includes: 1) adding these fields, 2) joining by these fields with all the source CTEs containing the source models with metrics - which in turn needs the change of the dates model - and 3) joining by these fields in the self-join to compute the incremental vs. previous year. * Changes on the schema file **2 - Intermediate, `int_mtd_aggregated_metrics`:** * Adds `dimension` and `dimension_value` granularity. This includes only adding these fields. * Changes on the schema file **3 - Reporting, `mtd_aggregated_metrics`:** * Adds the filter removed on `int_mtd_vs_previous_year_metrics`. This ensures that only the Global dimension is available for the reporting, thus **no changes from user POV**. * Adds `dimension` and `dimension_value` granularity. This includes only adding these fields * Changes on the schema file # Checklist - [X] The edited models and dependants run properly with production data. - [X] The edited models are sufficiently documented. - [X] The edited models contain PK tests, and I've ran and passed them. - [X] I have checked for DRY opportunities with other models and docs. - [X] I've picked the right materialization for the affected models. # Other - [ ] Check if a full-refresh is required after this PR is merged. Related work items: #19325
2024-08-20 15:42:27 +00:00
- name: dimension
data_type: string
description: The dimension or granularity of the metrics.
tests:
- accepted_values:
2024-09-12 15:38:50 +02:00
values:
Merged PR 2607: Propagates and exposes multiple dimension handling for KPIs # Description This PR ensures the propagation of the dimensions for KPIs across the key aggregating and exposing models. Additionally, provides these 2 new fields in reporting while **not affecting the current data display**, thus it's safe to work in the PBI report without needing to work in 2 PRs in parallel. **Changes:** **1 - Intermediate, `int_mtd_vs_previous_year_metrics`:** * Removes the temporary filter on `where dimension in ({{ production_dimensions }})`. This will be applied directly to reporting later. This ensures that the new dimension on customer segmentation is fully available only within intermediate. * Adds `dimension` and `dimension_value` granularity. This includes: 1) adding these fields, 2) joining by these fields with all the source CTEs containing the source models with metrics - which in turn needs the change of the dates model - and 3) joining by these fields in the self-join to compute the incremental vs. previous year. * Changes on the schema file **2 - Intermediate, `int_mtd_aggregated_metrics`:** * Adds `dimension` and `dimension_value` granularity. This includes only adding these fields. * Changes on the schema file **3 - Reporting, `mtd_aggregated_metrics`:** * Adds the filter removed on `int_mtd_vs_previous_year_metrics`. This ensures that only the Global dimension is available for the reporting, thus **no changes from user POV**. * Adds `dimension` and `dimension_value` granularity. This includes only adding these fields * Changes on the schema file # Checklist - [X] The edited models and dependants run properly with production data. - [X] The edited models are sufficiently documented. - [X] The edited models contain PK tests, and I've ran and passed them. - [X] I have checked for DRY opportunities with other models and docs. - [X] I've picked the right materialization for the affected models. # Other - [ ] Check if a full-refresh is required after this PR is merged. Related work items: #19325
2024-08-20 15:42:27 +00:00
- global
- by_number_of_listings
Merged PR 2689: KPIs by Billing Country # Description Adds Billing Country dimension in KPIs, but does not expose them to reporting yet. Silly thing, based on the macros I built, I cannot make incremental changes unless changing all models. This will need to be adapted, happy to hear your thoughts on how we do it. Additionally, I have lack of performance of the model `mtd_guest_payments_metrics`. It takes around 5 min to execute, but technically the end-to-end runs in one shoot without breaking. It's a complex PR because it changes many files, but you will see that: * It mostly changes the join conditions for the dimensions or the schema tests, * I tried to be very careful and add things step-by-step in the commits. Goal is NOT to complete the PR yet until we see how we can improve performance. I can say though that data end-to-end looks ok to me, but would benefit from checking with production data for the new dimension Update 30th Aug * Added a new commit that includes `id_user_host` in `int_core__verification_payments`. Happy to discuss if it makes sense or not. But it changes the execution from ~600 sec to ~6 sec because it avoids a massive repeated join with `verification_requests`. # Checklist - [X] The edited models and dependants run properly with production data. - [X] The edited models are sufficiently documented. - [X] The edited models contain PK tests, and I've ran and passed them. - [X] I have checked for DRY opportunities with other models and docs. - [ ] I've picked the right materialization for the affected models. **To check because of performance issues** # Other - [ ] Check if a full-refresh is required after this PR is merged. Related work items: #19082
2024-09-04 10:17:12 +00:00
- by_billing_country
2024-09-12 15:38:50 +02:00
Merged PR 2607: Propagates and exposes multiple dimension handling for KPIs # Description This PR ensures the propagation of the dimensions for KPIs across the key aggregating and exposing models. Additionally, provides these 2 new fields in reporting while **not affecting the current data display**, thus it's safe to work in the PBI report without needing to work in 2 PRs in parallel. **Changes:** **1 - Intermediate, `int_mtd_vs_previous_year_metrics`:** * Removes the temporary filter on `where dimension in ({{ production_dimensions }})`. This will be applied directly to reporting later. This ensures that the new dimension on customer segmentation is fully available only within intermediate. * Adds `dimension` and `dimension_value` granularity. This includes: 1) adding these fields, 2) joining by these fields with all the source CTEs containing the source models with metrics - which in turn needs the change of the dates model - and 3) joining by these fields in the self-join to compute the incremental vs. previous year. * Changes on the schema file **2 - Intermediate, `int_mtd_aggregated_metrics`:** * Adds `dimension` and `dimension_value` granularity. This includes only adding these fields. * Changes on the schema file **3 - Reporting, `mtd_aggregated_metrics`:** * Adds the filter removed on `int_mtd_vs_previous_year_metrics`. This ensures that only the Global dimension is available for the reporting, thus **no changes from user POV**. * Adds `dimension` and `dimension_value` granularity. This includes only adding these fields * Changes on the schema file # Checklist - [X] The edited models and dependants run properly with production data. - [X] The edited models are sufficiently documented. - [X] The edited models contain PK tests, and I've ran and passed them. - [X] I have checked for DRY opportunities with other models and docs. - [X] I've picked the right materialization for the affected models. # Other - [ ] Check if a full-refresh is required after this PR is merged. Related work items: #19325
2024-08-20 15:42:27 +00:00
- name: dimension_value
data_type: string
description: The value or segment available for the selected dimension.
tests:
- not_null
- name: int_mtd_aggregated_metrics
description: |
The `int_mtd_aggregated_metrics` model aggregates multiple metrics on a year, month, and day basis.
The primary source of data is the `int_mtd_vs_previous_year_metrics` model, which contain the combination
of metrics data per source. This model just changes the display format to unpivot the information into
a set of metric, value, previous_year_value and relative_increment at a given date. It uses Jinja
code to avoid code replication.
2024-09-12 15:38:50 +02:00
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- date
- metric
Merged PR 2607: Propagates and exposes multiple dimension handling for KPIs # Description This PR ensures the propagation of the dimensions for KPIs across the key aggregating and exposing models. Additionally, provides these 2 new fields in reporting while **not affecting the current data display**, thus it's safe to work in the PBI report without needing to work in 2 PRs in parallel. **Changes:** **1 - Intermediate, `int_mtd_vs_previous_year_metrics`:** * Removes the temporary filter on `where dimension in ({{ production_dimensions }})`. This will be applied directly to reporting later. This ensures that the new dimension on customer segmentation is fully available only within intermediate. * Adds `dimension` and `dimension_value` granularity. This includes: 1) adding these fields, 2) joining by these fields with all the source CTEs containing the source models with metrics - which in turn needs the change of the dates model - and 3) joining by these fields in the self-join to compute the incremental vs. previous year. * Changes on the schema file **2 - Intermediate, `int_mtd_aggregated_metrics`:** * Adds `dimension` and `dimension_value` granularity. This includes only adding these fields. * Changes on the schema file **3 - Reporting, `mtd_aggregated_metrics`:** * Adds the filter removed on `int_mtd_vs_previous_year_metrics`. This ensures that only the Global dimension is available for the reporting, thus **no changes from user POV**. * Adds `dimension` and `dimension_value` granularity. This includes only adding these fields * Changes on the schema file # Checklist - [X] The edited models and dependants run properly with production data. - [X] The edited models are sufficiently documented. - [X] The edited models contain PK tests, and I've ran and passed them. - [X] I have checked for DRY opportunities with other models and docs. - [X] I've picked the right materialization for the affected models. # Other - [ ] Check if a full-refresh is required after this PR is merged. Related work items: #19325
2024-08-20 15:42:27 +00:00
- dimension
- dimension_value
2024-09-12 15:38:50 +02:00
columns:
- name: year
data_type: int
description: year number of the given date.
tests:
- not_null
- name: month
2024-09-12 15:38:50 +02:00
data_type: int
description: month number of the given date.
tests:
- not_null
- name: day
data_type: int
description: day monthly number of the given date.
tests:
- not_null
- name: is_end_of_month
data_type: boolean
description: True if it's end of month.
tests:
- not_null
- name: is_end_of_month_or_yesterday
data_type: boolean
description: True if it's end of month or yesterday.
tests:
- not_null
- name: is_current_month
data_type: boolean
description: |
checks if the date is within the current executed month,
1 for yes, 0 for no.
tests:
- not_null
- name: first_day_month
data_type: date
description: |
first day of the month corresponding to the date field.
tests:
- not_null
- name: date
data_type: date
description: |
main date for the computation, that is used for filters.
tests:
- not_null
Merged PR 2607: Propagates and exposes multiple dimension handling for KPIs # Description This PR ensures the propagation of the dimensions for KPIs across the key aggregating and exposing models. Additionally, provides these 2 new fields in reporting while **not affecting the current data display**, thus it's safe to work in the PBI report without needing to work in 2 PRs in parallel. **Changes:** **1 - Intermediate, `int_mtd_vs_previous_year_metrics`:** * Removes the temporary filter on `where dimension in ({{ production_dimensions }})`. This will be applied directly to reporting later. This ensures that the new dimension on customer segmentation is fully available only within intermediate. * Adds `dimension` and `dimension_value` granularity. This includes: 1) adding these fields, 2) joining by these fields with all the source CTEs containing the source models with metrics - which in turn needs the change of the dates model - and 3) joining by these fields in the self-join to compute the incremental vs. previous year. * Changes on the schema file **2 - Intermediate, `int_mtd_aggregated_metrics`:** * Adds `dimension` and `dimension_value` granularity. This includes only adding these fields. * Changes on the schema file **3 - Reporting, `mtd_aggregated_metrics`:** * Adds the filter removed on `int_mtd_vs_previous_year_metrics`. This ensures that only the Global dimension is available for the reporting, thus **no changes from user POV**. * Adds `dimension` and `dimension_value` granularity. This includes only adding these fields * Changes on the schema file # Checklist - [X] The edited models and dependants run properly with production data. - [X] The edited models are sufficiently documented. - [X] The edited models contain PK tests, and I've ran and passed them. - [X] I have checked for DRY opportunities with other models and docs. - [X] I've picked the right materialization for the affected models. # Other - [ ] Check if a full-refresh is required after this PR is merged. Related work items: #19325
2024-08-20 15:42:27 +00:00
- name: dimension
data_type: string
description: The dimension or granularity of the metrics.
tests:
- accepted_values:
2024-09-12 15:38:50 +02:00
values:
Merged PR 2607: Propagates and exposes multiple dimension handling for KPIs # Description This PR ensures the propagation of the dimensions for KPIs across the key aggregating and exposing models. Additionally, provides these 2 new fields in reporting while **not affecting the current data display**, thus it's safe to work in the PBI report without needing to work in 2 PRs in parallel. **Changes:** **1 - Intermediate, `int_mtd_vs_previous_year_metrics`:** * Removes the temporary filter on `where dimension in ({{ production_dimensions }})`. This will be applied directly to reporting later. This ensures that the new dimension on customer segmentation is fully available only within intermediate. * Adds `dimension` and `dimension_value` granularity. This includes: 1) adding these fields, 2) joining by these fields with all the source CTEs containing the source models with metrics - which in turn needs the change of the dates model - and 3) joining by these fields in the self-join to compute the incremental vs. previous year. * Changes on the schema file **2 - Intermediate, `int_mtd_aggregated_metrics`:** * Adds `dimension` and `dimension_value` granularity. This includes only adding these fields. * Changes on the schema file **3 - Reporting, `mtd_aggregated_metrics`:** * Adds the filter removed on `int_mtd_vs_previous_year_metrics`. This ensures that only the Global dimension is available for the reporting, thus **no changes from user POV**. * Adds `dimension` and `dimension_value` granularity. This includes only adding these fields * Changes on the schema file # Checklist - [X] The edited models and dependants run properly with production data. - [X] The edited models are sufficiently documented. - [X] The edited models contain PK tests, and I've ran and passed them. - [X] I have checked for DRY opportunities with other models and docs. - [X] I've picked the right materialization for the affected models. # Other - [ ] Check if a full-refresh is required after this PR is merged. Related work items: #19325
2024-08-20 15:42:27 +00:00
- global
- by_number_of_listings
Merged PR 2689: KPIs by Billing Country # Description Adds Billing Country dimension in KPIs, but does not expose them to reporting yet. Silly thing, based on the macros I built, I cannot make incremental changes unless changing all models. This will need to be adapted, happy to hear your thoughts on how we do it. Additionally, I have lack of performance of the model `mtd_guest_payments_metrics`. It takes around 5 min to execute, but technically the end-to-end runs in one shoot without breaking. It's a complex PR because it changes many files, but you will see that: * It mostly changes the join conditions for the dimensions or the schema tests, * I tried to be very careful and add things step-by-step in the commits. Goal is NOT to complete the PR yet until we see how we can improve performance. I can say though that data end-to-end looks ok to me, but would benefit from checking with production data for the new dimension Update 30th Aug * Added a new commit that includes `id_user_host` in `int_core__verification_payments`. Happy to discuss if it makes sense or not. But it changes the execution from ~600 sec to ~6 sec because it avoids a massive repeated join with `verification_requests`. # Checklist - [X] The edited models and dependants run properly with production data. - [X] The edited models are sufficiently documented. - [X] The edited models contain PK tests, and I've ran and passed them. - [X] I have checked for DRY opportunities with other models and docs. - [ ] I've picked the right materialization for the affected models. **To check because of performance issues** # Other - [ ] Check if a full-refresh is required after this PR is merged. Related work items: #19082
2024-09-04 10:17:12 +00:00
- by_billing_country
2024-09-12 15:38:50 +02:00
Merged PR 2607: Propagates and exposes multiple dimension handling for KPIs # Description This PR ensures the propagation of the dimensions for KPIs across the key aggregating and exposing models. Additionally, provides these 2 new fields in reporting while **not affecting the current data display**, thus it's safe to work in the PBI report without needing to work in 2 PRs in parallel. **Changes:** **1 - Intermediate, `int_mtd_vs_previous_year_metrics`:** * Removes the temporary filter on `where dimension in ({{ production_dimensions }})`. This will be applied directly to reporting later. This ensures that the new dimension on customer segmentation is fully available only within intermediate. * Adds `dimension` and `dimension_value` granularity. This includes: 1) adding these fields, 2) joining by these fields with all the source CTEs containing the source models with metrics - which in turn needs the change of the dates model - and 3) joining by these fields in the self-join to compute the incremental vs. previous year. * Changes on the schema file **2 - Intermediate, `int_mtd_aggregated_metrics`:** * Adds `dimension` and `dimension_value` granularity. This includes only adding these fields. * Changes on the schema file **3 - Reporting, `mtd_aggregated_metrics`:** * Adds the filter removed on `int_mtd_vs_previous_year_metrics`. This ensures that only the Global dimension is available for the reporting, thus **no changes from user POV**. * Adds `dimension` and `dimension_value` granularity. This includes only adding these fields * Changes on the schema file # Checklist - [X] The edited models and dependants run properly with production data. - [X] The edited models are sufficiently documented. - [X] The edited models contain PK tests, and I've ran and passed them. - [X] I have checked for DRY opportunities with other models and docs. - [X] I've picked the right materialization for the affected models. # Other - [ ] Check if a full-refresh is required after this PR is merged. Related work items: #19325
2024-08-20 15:42:27 +00:00
- name: dimension_value
data_type: string
description: The value or segment available for the selected dimension.
tests:
- not_null
- name: previous_year_date
data_type: date
description: |
corresponds to the date of the previous year, with respect to the field date.
It's only displayed for information purposes,
should not be needed for reporting.
- name: metric
data_type: text
description: name of the business metric.
tests:
- not_null
- name: order_by
data_type: integer
description: |
order for displaying purposes. Null values are accepted, but keep
in mind that then there's no default controlled display order.
- name: number_format
data_type: text
description: allows for grouping and formatting for displaying purposes.
tests:
2024-09-12 15:38:50 +02:00
- accepted_values:
2024-09-20 14:53:43 +02:00
values:
[
"integer",
"percentage",
"currency_gbp",
"converted_metric_currency_gbp",
]
- name: value
2024-09-12 15:38:50 +02:00
data_type: numeric
description: |
numeric value (integer or decimal) that corresponds to the MTD computation of the metric
at a given date.
- name: previous_year_value
2024-09-12 15:38:50 +02:00
data_type: numeric
description: |
numeric value (integer or decimal) that corresponds to the MTD computation of the metric
on the previous year at a given date.
- name: relative_increment
2024-09-12 15:38:50 +02:00
data_type: numeric
description: |
numeric value that corresponds to the relative increment between value and previous year value,
following the computation: value / previous_year_value - 1.
- name: relative_increment_with_sign_format
2024-09-12 15:38:50 +02:00
data_type: numeric
description: |
relative_increment value multiplied by -1 in case this metric's growth doesn't have a
positive impact for Superhog, otherwise is equal to relative_increment.
This value is specially created for formatting in PBI
- name: int_monthly_aggregated_metrics_history_by_deal
description: |
This model aggregates the monthly historic information regarding the different metrics computed
at deal level. The primary sources of data are the `int_yyy__monthly_XXXXX_history_by_deal`
models which contain the raw metrics data per source.
Unlike the int_mtd_aggregated_metrics, this model does not abstract each metric, since
no comparison versus last year is performed. In short, it just gathers the information stored
in the abovementioned models.
To keep in mind: aggregating the information of this model will not necessarily result into
the int_mtd_aggregated metrics because 1) the mtd version contains more computing dates
than the by deal version, the latest being a subset of the first, and 2) the deal based model
enforces that a booking/guest journey/listing/etc has a host with a deal assigned, which is
2024-09-12 15:38:50 +02:00
not necessarily the case.
2024-09-12 15:38:50 +02:00
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- date
- id_deal
columns:
- name: date
data_type: date
description: The last day of the month or yesterday for historic metrics.
tests:
- not_null
- name: id_deal
data_type: character varying
2024-09-12 15:38:50 +02:00
description: Id of the deal associated to the host.
tests:
- not_null
- name: main_deal_name
data_type: string
description: |
Main name for this ID deal.
tests:
- not_null
2025-01-09 16:31:22 +01:00
- name: has_active_pms
data_type: boolean
description: |
Does the deal have an active associated PMS.
tests:
- not_null
2025-01-09 16:31:22 +01:00
- name: active_pms_list
data_type: string
description: |
Name of the active PMS associated with the deal. It can have more than
2025-01-09 16:31:22 +01:00
one PMS associated with it. It can be null if it doesn't have any PMS associated.
- name: main_billing_country_iso_3_per_deal
data_type: string
description: |
ISO 3166-1 alpha-3 main country code in which the Deal is billed.
In some cases it's null.
2024-09-12 15:38:50 +02:00
- name: int_monthly_growth_score_by_deal
description: |
The main goal of this model is to provide a growth score by deal and month.
The idea behind it is that each deal will have some business performance
associated to it over the months, and that comparing how it is currently
performing vs. historical data we can determine whether the tendency is to
grow or to decay. This is specially useful for AMs to focus their effort
towards the clients that have a negative tendency.
The computation of the growth score is based on 3 main indicators:
- Created bookings
- Listings booked in month
- Total revenue (in gbp)
The main idea is, for each deal, to compare each of these metrics by
checking the latest monthly value vs. 1) the monthly value of the equivalent
month on the previous year and 2) the monthly value of the previous month
- in other words, a year-on-year (YoY) and month-on-month (MoM) comparison.
We do this comparison by doing a relative incremental.
The growth score is computed then by averaging the outcome of the 6 scores.
Lastly, in order to provide a prioritisation sense, we have a weighted growth
score that results from the multiplication of the growth score per the revenue
weight a specific deal has provided in the previous 12 months.
However, this is not strictly true for Revenue because 1) we have an invoicing
delay and 2) in some cases, monthly revenue per deal can be negative. In this
specific cases, the YoY comparison is shifted by one month, and an effective
revenue value for the revenue share is computed, that cannot be lower than 0.
In order to keep both a properly set up score and revenue consistency, both
a real revenue value and effective revenue value are present in this model,
while no MoM or YoY value is computed if negative revenue is found.
Lastly, this model provides informative date fields, deal attributes, absolute
metric values and MoM & YoY relative incrementals to enrich reporting.
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- date
- id_deal
columns:
- name: date
data_type: date
description: |
Date corresponding to the last day of the month. Given month
metrics are inclusive to this date. Together with id_deal, it
acts as the primary key of this model.
tests:
- not_null
- name: id_deal
data_type: string
description: |
Unique identifier of a Deal. Together with date, it acts as
the primary key of this model.
tests:
- not_null
- name: main_deal_name
data_type: string
description: |
Main name for a Deal, representing the client.
tests:
- not_null
2025-01-10 09:17:29 +01:00
- name: has_active_pms
data_type: boolean
description: |
Does the deal have an active associated PMS.
tests:
- not_null
- name: active_pms_list
data_type: string
description: |
Name of the active PMS associated with the deal. It can have more than
one PMS associated with it. It can be null if it doesn't have any PMS associated.
- name: main_billing_country_iso_3_per_deal
data_type: string
description: |
Main billing country for this client. In some cases
it can be null.
- name: deal_lifecycle_state
data_type: string
description: |
Identifier of the lifecycle state of a given deal
in a given month.
- name: deal_hubspot_stage
data_type: string
description: |
Current hubspot stage for a given deal.
- name: account_manager
data_type: string
description: |
Current Account Manager in charge of a given deal, according
to Hubspot.
- name: live_date_utc
data_type: date
description: |
Date in which the account has gone live, according to Hubspot.
- name: cancellation_date_utc
data_type: date
description: |
Date in which the account has been offboarded, according to
Hubspot.
- name: given_month_first_day_month
data_type: date
description: |
Informative field. It indicates the first day of the
month corresponding to date.
If date = 2024-09-30, this field will be 2024-09-01.
tests:
- not_null
- name: previous_1_month_first_day_month
data_type: date
description: |
Informative field. It indicates the first day of the
previous month with respect to date.
If date = 2024-09-30, this field will be 2024-08-01.
It can be null if no previous history for that
deal is found.
- name: previous_2_month_first_day_month
data_type: date
description: |
Informative field. It indicates the first day of the
month 2 months before with respect to date.
If date = 2024-09-30, this field will be 2024-07-01.
It can be null if no previous history for that
deal is found.
- name: previous_12_month_first_day_month
data_type: date
description: |
Informative field. It indicates the first day of the
month with respect to date, but on the previous year.
If date = 2024-09-30, this field will be 2023-09-01.
It can be null if no previous history for that
deal is found.
- name: previous_13_month_first_day_month
data_type: date
description: |
Informative field. It indicates the first day of the
previous month with respect to date, but on the previous year.
If date = 2024-09-30, this field will be 2023-08-01.
It can be null if no previous history for that
deal is found.
- name: aggregated_revenue_from_first_day_month
data_type: date
description: |
Informative field. It indicates the first day of the
month from the lower bound range in which the revenue
aggregation is computed.
The aggregation uses the previous 12 months in which we
know the revenue, thus:
If date = 2024-09-30, this field will be 2023-09-01.
It can be null if no previous history for that
deal is found.
- name: aggregated_revenue_to_first_day_month
data_type: date
description: |
Informative field. It indicates the first day of the
month from the upper bound range in which the revenue
aggregation is computed.
The aggregation uses the previous 12 months in which we
know the revenue, thus:
If date = 2024-09-30, this field will be 2023-08-01.
It can be null if no previous history for that
deal is found.
- name: given_month_revenue_in_gbp
data_type: decimal
description: |
Monthly value representing revenue in GBP
for a specific deal. This value corresponds to
the given month. This value can be negative,
but not null.
tests:
- not_null
- name: previous_1_month_revenue_in_gbp
data_type: decimal
description: |
Monthly value representing revenue in GBP
for a specific deal. This value corresponds to
the previous month.
This value can be negative.
This value can be null, thus indicating that no
history is available.
- name: previous_2_month_revenue_in_gbp
data_type: decimal
description: |
Monthly value representing revenue in GBP
for a specific deal. This value corresponds to
the monthly amount generated 2 months ago
This value can be negative.
This value can be null, thus indicating that no
history is available.
- name: previous_12_month_revenue_in_gbp
data_type: decimal
description: |
Monthly value representing revenue in GBP
for a specific deal. This value corresponds to
the monthly amount generated 12 months ago.
This value can be negative.
This value can be null, thus indicating that no
history is available.
- name: previous_13_month_revenue_in_gbp
data_type: decimal
description: |
Monthly value representing revenue in GBP
for a specific deal. This value corresponds to
the monthly amount generated 13 months ago.
This value can be negative.
This value can be null, thus indicating that no
history is available.
- name: mom_revenue_growth
data_type: decimal
description: |
Relative increment of the revenue generated in the
current month with respect to the one generated in
the previous month.
It can be null if any revenue used in the computation
is null or it's negative.
tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: -1
strictly: false
- name: mom_1_month_shift_revenue_growth
data_type: decimal
description: |
Relative increment of the revenue generated in the
previous month with respect to the one generated 2
months ago.
It can be null if any revenue used in the computation
is null or it's negative.
This field is used for the growth score computation.
tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: -1
strictly: false
- name: yoy_revenue_growth
data_type: decimal
description: |
Relative increment of the revenue generated in the
current month with respect to the one generated 12
months ago.
It can be null if any revenue used in the computation
is null or it's negative.
tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: -1
strictly: false
- name: yoy_1_month_shift_revenue_growth
data_type: decimal
description: |
Relative increment of the revenue generated in the
previous month with respect to the one generated 13
months ago.
It can be null if any revenue used in the computation
is null or it's negative.
This field is used for the growth score computation.
tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: -1
strictly: false
- name: given_month_created_bookings
data_type: integer
description: |
Monthly value representing created bookings
for a specific deal. This value corresponds to
the given month. This value cannot be null.
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: previous_1_month_created_bookings
data_type: integer
description: |
Monthly value representing created bookings
for a specific deal. This value corresponds to
the previous month.
This value can be null, thus indicating that no
history is available.
tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: previous_12_month_created_bookings
data_type: integer
description: |
Monthly value representing created bookings
for a specific deal. This value corresponds to
monthly amount generated 12 months ago.
This value can be null, thus indicating that no
history is available.
tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: mom_created_bookings_growth
data_type: decimal
description: |
Relative increment of the bookings created in the
current month with respect to the ones created in
the previous month.
It can be null if the bookings created in the
previous month are null.
This field is used for the growth score computation.
tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: -1
strictly: false
- name: yoy_created_bookings_growth
data_type: decimal
description: |
Relative increment of the bookings created in the
current month with respect to the ones created 12
months ago.
It can be null if the bookings created 12 months
ago are null.
This field is used for the growth score computation.
tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: -1
strictly: false
- name: given_month_listings_booked_in_month
data_type: integer
description: |
Monthly value representing the listings booked in month
for a specific deal. This value corresponds to
the given month. This value cannot be null.
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: previous_1_month_listings_booked_in_month
data_type: integer
description: |
Monthly value representing the listings booked in month
for a specific deal. This value corresponds to
the previous month.
This value can be null, thus indicating that no
history is available.
tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: previous_12_month_listings_booked_in_month
data_type: integer
description: |
Monthly value representing the listings booked in month
for a specific deal. This value corresponds to
monthly amount generated 12 months ago.
This value can be null, thus indicating that no
history is available.
tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: mom_listings_booked_in_month_growth
data_type: decimal
description: |
Relative increment of the the listings booked in month
in the current month with respect to the ones of
the previous month.
It can be null if the listings booked in month in the
previous month are null.
This field is used for the growth score computation.
tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: -1
strictly: false
- name: yoy_listings_booked_in_month_growth
data_type: decimal
description: |
Relative increment of the listings booked in month
in the current month with respect to the ones of 12
months ago.
It can be null if the listings booked in month of 12
months ago are null.
This field is used for the growth score computation.
tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: -1
strictly: false
- name: deal_revenue_12_months_window
data_type: decimal
description: |
Total aggregated revenue in GBP generated by a deal
in the months from the period ranging from the
aggregated_revenue_from_first_day_month to
aggregated_revenue_to_first_day_month.
It can be negative if the sum is negative.
It cannot be null.
tests:
- not_null
- name: effective_deal_revenue_12_months_window
data_type: decimal
description: |
Effective aggregated revenue in GBP generated by a deal
in the months from the period ranging from the
aggregated_revenue_from_first_day_month to
aggregated_revenue_to_first_day_month.
All negative monthly revenue values are settled as 0,
thus this value should not be reported.
It is used for the deal contribution share with respect
to the global revenue. It cannot be null.
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: effective_global_revenue_12_months_window
data_type: decimal
description: |
Effective aggregated revenue in GBP generated by all deals
in the months from the period ranging from the
aggregated_revenue_from_first_day_month to
aggregated_revenue_to_first_day_month.
All negative monthly revenue values are settled as 0,
thus this value should not be reported.
It is used for the deal contribution share with respect
to the global revenue. It cannot be null.
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: deal_contribution_share_to_global_revenue
data_type: decimal
description: |
Represents the size of the deal in terms of revenue. In
other words, what's the percentage of the global revenue
that can be attributed to this deal. It cannot be null.
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: deal_contribution_rank_to_global_revenue
data_type: integer
description: |
Represents the ordered list of deals by descending size
in terms of revenue.
If more than one deal have the same share, the order is
not under control.
It cannot be null.
tests:
- not_null
- name: deal_created_bookings_12_months_window
data_type: integer
description: |
Total created bookings generated by a deal
in the months from the period ranging from the
aggregated_revenue_from_first_day_month to
aggregated_revenue_to_first_day_month.
It cannot be null.
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: global_created_bookings_12_months_window
data_type: integer
description: |
Total created bookings generated by any deal
in the months from the period ranging from the
aggregated_revenue_from_first_day_month to
aggregated_revenue_to_first_day_month.
It is used for the deal contribution share with respect
to the global created bookings. It cannot be null.
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: deal_contribution_share_to_global_created_bookings
data_type: decimal
description: |
Represents the size of the deal in terms of created bookings.
In other words, what's the percentage of the global created
bookings that can be attributed to this deal.
It cannot be null.
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: deal_contribution_rank_to_global_created_bookings
data_type: integer
description: |
Represents the ordered list of deals by descending size
in terms of created bookings.
If more than one deal have the same share, the order is
not under control.
It cannot be null.
tests:
- not_null
- name: deal_avg_listings_booked_in_month_12_months_window
data_type: decimal
description: |
Average listings booked in month by a deal
in the months from the period ranging from the
aggregated_revenue_from_first_day_month to
aggregated_revenue_to_first_day_month.
It cannot be null.
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: global_avg_listings_booked_in_month_12_months_window
data_type: decimal
description: |
Sum of the average listings booked in month by
any deal in the months from the period ranging from the
aggregated_revenue_from_first_day_month to
aggregated_revenue_to_first_day_month.
It is used for the deal contribution share with respect
to the global average listings booked in month.
It cannot be null.
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: deal_contribution_share_to_global_avg_listings_booked_in_month
data_type: decimal
description: |
Represents the size of the deal in terms of average listings
booked in month.
In other words, what's the percentage of the global average listings
booked in month that can be attributed to this deal.
It cannot be null.
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: deal_contribution_rank_to_global_avg_listings_booked_in_month
data_type: decimal
description: |
Represents the ordered list of deals by descending size
in terms of average listings booked in month.
If more than one deal have the same share, the order is
not under control.
It cannot be null.
tests:
- not_null
- name: avg_mom_growth_score
data_type: decimal
description: |
Represents the average score of MoM growth of created
bookings, MoM growth of listings booked in month and
MoM shifted by one month of revenue.
It indicates the tendency of growth of the deal without
taking into account its revenue size. It cannot be null.
tests:
- not_null
- name: avg_yoy_growth_score
data_type: decimal
description: |
Represents the average score of YoY growth of created
bookings, YoY growth of listings booked in month and
YoY shifted by one month of revenue.
It indicates the tendency of growth of the deal without
taking into account its revenue size. It cannot be null.
tests:
- not_null
- name: avg_growth_score
data_type: decimal
description: |
Represents the average score of YoY and MoM growth of created
bookings, YoY and MoM growth of listings booked in month and
YoY and MoM shifted by one month of revenue.
It indicates the tendency of growth of the deal without
taking into account its revenue size. It cannot be null.
tests:
- not_null
- name: weighted_avg_growth_score
data_type: decimal
description: |
It's the weighted version of avg_growth_score that
takes into account the client size by using the revenue
contribution share of that deal to the global amount.
It's the main indicator towards measuring both growth
(if positive) or decay (if negative) while weighting
the financial impact this deal tendency can have.
tests:
- not_null
- name: categorisation_weighted_avg_growth_score
data_type: string
description: |
Discrete categorisation of weighted_avg_growth_score.
It helps easily identifying which accounts are top losers,
losers, flat, winners and top winners.
Currently the categorisation is based on the score itself
rather than selecting a top up/down.
tests:
- not_null
- accepted_values:
values:
- MAJOR DECLINE
- DECLINE
- FLAT
- GAIN
- MAJOR GAIN
- UNSET
Merged PR 3163: First version of 12m window contribution by deal # Description This PR creates a new model that depends on int_monthly_aggregated_metrics_history_by_deal. The idea is that this is used for Churn computation (Booking Churn, Revenue Churn, Listing Churn) later on. The idea is relatively simple. Measure how much a Deal has been contributing to a Global amount (sum of metric for all deals) over the preceding period of 12 months. You will notice that there's 2 computations, the "additive" and the "average" one. This is because we still need to align with Matt/Suzannah on which approach makes more sense, but we need data for it. I'm not sure the namings are good though so happy to see your suggestions. You will also notice that there's no filter by deal_lifecycle_state = '06-Churning'. This will be done in a separated model, whenever we attribute this model to the mtd computation. The reason is simple - this model stays at deal level, thus meaning we can do the dimension aggregation and even a lifecycle aggregation if needed, depending on the needs. Be aware that this effectively means that MTD KPIs models will depend on the "monthly by deal" models. This has some cons in terms of dependency management but cannot be overcome since we the metric total revenue depends on many subsets. In essence, I don't see another way of doing it unless doing a massive KPIs refactor. I prefer to wait until the Product KPIs discussions are finished and then we see how we approach it. # Checklist - [X] The edited models and dependants run properly with production data. - [X] The edited models are sufficiently documented. - [X] The edited models contain PK tests, and I've ran and passed them. - [X] I have checked for DRY opportunities with other models and docs. - [X] I've picked the right materialization for the affected models. # Other - [ ] Check if a full-refresh is required after this PR is merged. Related work items: #22691
2024-10-15 06:51:41 +00:00
- name: int_monthly_12m_window_contribution_by_deal
description: |
The main goal of this model is to provide how much a deal
contributes to a given metric on the global amount over a
period of 12 months.
At the moment, this is only done for 3 metrics:
- total_revenue_in_gbp
- created_bookings
- listings_booked_in_month
The contribution is based on an Average approach:
Merged PR 3163: First version of 12m window contribution by deal # Description This PR creates a new model that depends on int_monthly_aggregated_metrics_history_by_deal. The idea is that this is used for Churn computation (Booking Churn, Revenue Churn, Listing Churn) later on. The idea is relatively simple. Measure how much a Deal has been contributing to a Global amount (sum of metric for all deals) over the preceding period of 12 months. You will notice that there's 2 computations, the "additive" and the "average" one. This is because we still need to align with Matt/Suzannah on which approach makes more sense, but we need data for it. I'm not sure the namings are good though so happy to see your suggestions. You will also notice that there's no filter by deal_lifecycle_state = '06-Churning'. This will be done in a separated model, whenever we attribute this model to the mtd computation. The reason is simple - this model stays at deal level, thus meaning we can do the dimension aggregation and even a lifecycle aggregation if needed, depending on the needs. Be aware that this effectively means that MTD KPIs models will depend on the "monthly by deal" models. This has some cons in terms of dependency management but cannot be overcome since we the metric total revenue depends on many subsets. In essence, I don't see another way of doing it unless doing a massive KPIs refactor. I prefer to wait until the Product KPIs discussions are finished and then we see how we approach it. # Checklist - [X] The edited models and dependants run properly with production data. - [X] The edited models are sufficiently documented. - [X] The edited models contain PK tests, and I've ran and passed them. - [X] I have checked for DRY opportunities with other models and docs. - [X] I've picked the right materialization for the affected models. # Other - [ ] Check if a full-refresh is required after this PR is merged. Related work items: #22691
2024-10-15 06:51:41 +00:00
Over a period of 12 months, sum the value of a given a metric
for each deal, and divide it by the amount of months we're considering
for that deal. Sum all the average amounts per deals to get a global.
Divide the avg per deal value vs. the sum of avgs global one.
The average approach "boosts" the contribution of those accounts
that have been active for less than 12 months.
Merged PR 3163: First version of 12m window contribution by deal # Description This PR creates a new model that depends on int_monthly_aggregated_metrics_history_by_deal. The idea is that this is used for Churn computation (Booking Churn, Revenue Churn, Listing Churn) later on. The idea is relatively simple. Measure how much a Deal has been contributing to a Global amount (sum of metric for all deals) over the preceding period of 12 months. You will notice that there's 2 computations, the "additive" and the "average" one. This is because we still need to align with Matt/Suzannah on which approach makes more sense, but we need data for it. I'm not sure the namings are good though so happy to see your suggestions. You will also notice that there's no filter by deal_lifecycle_state = '06-Churning'. This will be done in a separated model, whenever we attribute this model to the mtd computation. The reason is simple - this model stays at deal level, thus meaning we can do the dimension aggregation and even a lifecycle aggregation if needed, depending on the needs. Be aware that this effectively means that MTD KPIs models will depend on the "monthly by deal" models. This has some cons in terms of dependency management but cannot be overcome since we the metric total revenue depends on many subsets. In essence, I don't see another way of doing it unless doing a massive KPIs refactor. I prefer to wait until the Product KPIs discussions are finished and then we see how we approach it. # Checklist - [X] The edited models and dependants run properly with production data. - [X] The edited models are sufficiently documented. - [X] The edited models contain PK tests, and I've ran and passed them. - [X] I have checked for DRY opportunities with other models and docs. - [X] I've picked the right materialization for the affected models. # Other - [ ] Check if a full-refresh is required after this PR is merged. Related work items: #22691
2024-10-15 06:51:41 +00:00
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- date
- id_deal
columns:
- name: date
data_type: date
description: |
Date corresponding to the last day of the month.
Metrics are inclusive to this date. Together with id_deal, it
acts as the primary key of this model.
tests:
- not_null
- name: id_deal
data_type: string
description: |
Unique identifier of a Deal. Together with date, it acts as
the primary key of this model.
tests:
- not_null
- name: deal_lifecycle_state
data_type: string
description: |
Identifier of the lifecycle state of a given deal
in a given month.
- name: preceding_months_count_by_deal
data_type: integer
description: |
Number of months preceding to the one given by date
that are used for the historic metric retrieval for
a given deal. In essence it states the amount of
months a given deal has been active before a the month
given by date, capped at 12 months.
tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
max_value: 12
strictly: false
- name: has_deal_been_created_less_than_12_months_ago
data_type: boolean
description: |
Flag to identify if a given deal has been created less
than 12 months ago (true) or not (false). It's based on the
preceding_months_count_by_deal, and will be true on the first
year of deal activity.
- name: total_revenue_12m_average_contribution
data_type: numeric
description: |
Share of the deal contribution on total revenue
vs. the global amount, on the preceding 12 months
with respect to date. It uses the average approach.
It can be negative.
tests:
- not_null
- name: created_bookings_12m_average_contribution
data_type: numeric
description: |
Share of the deal contribution on created bookings
vs. the global amount, on the preceding 12 months
with respect to date. It uses the average approach.
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
max_value: 1
strictly: false
- name: listings_booked_in_month_12m_average_contribution
data_type: numeric
description: |
Share of the deal contribution on listings booked in month
vs. the global amount, on the preceding 12 months
with respect to date. It uses the average approach.
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
max_value: 1
strictly: false
- name: int_monthly_churn_metrics
description: |
This model is used for global KPIs.
It computes the churn contribution by dimension, dimension value
and date, in a monthly basis. This model is different from the
usual mtd ones since it strictly depends on the monthly computation
of metrics by deal, which is done in a monthly basis rather than mtd.
In essence, it means we won't have data for the current month.
This model retrieves the 12 month contribution to global metrics
by deal and aggregates it to dimension and dimension value for those
deals that are tagged as '05-Churning' in that month. Thus, it provides
a total of 3 churn related metrics, represented as ratios over the total:
- Total Revenue (in GBP)
- Created Bookings
- Listings Booked in Month
by using the Average contribution method. For further
information, please refer to the documentation of the model:
- int_monthly_12m_window_contribution_by_deal
Lastly, when checking data at any dimension distinct from Global, at the
moment these values represent the additive contribution of churn with respect
to the global amount. This means that, for instance, if we have 10% of churn
in a month, it can be divided by 9% USA and 1% GBR since 9%+1% = 10%.
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- date
- dimension
- dimension_value
columns:
- name: date
data_type: date
description: The date for the month-to-date metrics.
tests:
- not_null
- name: dimension
data_type: string
description: The dimension or granularity of the metrics.
tests:
- accepted_values:
values:
- global
- by_number_of_listings
- by_billing_country
- name: dimension_value
data_type: string
description: The value or segment available for the selected dimension.
tests:
- not_null
- name: total_revenue_churn_average_contribution
data_type: numeric
description: Total Revenue churn rate (average approach).
- name: created_bookings_churn_average_contribution
data_type: numeric
description: Created Bookings churn rate (average approach).
- name: listings_booked_in_month_churn_average_contribution
data_type: numeric
description: Listings Booked in Month churn rate (average approach).
2024-11-29 09:59:35 +01:00
2024-11-29 11:03:59 +01:00
- name: int_edeposit_and_athena_verifications
2024-11-29 09:59:35 +01:00
description:
"This table holds records on verifications for Guesty and Edeposit bookings.
It contains details on validations checked on the guests, guest information
and some booking details like checkin-checkout date or the status of the verification.
The id values found here are completely unrelated to the ones found in Core DWH.
Note that id_verifications and booking_id should normally be 1 to 1.
Though there are exception, the API will accept a duplicate booking and the users
will be charged for it. A duplicate would return a unique id_verification."
columns:
- name: id_verification
data_type: text
description: "unique Superhog generated id for this verification"
tests:
- unique
- not_null
- name: id_booking
data_type: text
description: "unique Superhog generated id for a booking.
note that this could be duplicated and both will be charged,
it's up to the user to no generate duplicate verifications"
- name: id_user_partner
data_type: text
description: "unique Superhog generated id for partner"
tests:
- not_null
- name: id_accommodation
data_type: text
description: "unique Superhog generated id for a listing"
- name: version
data_type: text
description: "value to identify if it is Guesty (V1) or E-deposit (V2)"
tests:
- accepted_values:
values:
- V1
- V2
- name: verification_source
data_type: text
description: "source of the verification for the booking"
tests:
- accepted_values:
values:
- Guesty
- Edeposit
- name: verification_status
data_type: text
description: "status of the verification"
- name: nightly_fee_local
data_type: double precision
2024-12-03 10:37:05 +01:00
description: "fee in local currency charged per night"
2024-11-29 09:59:35 +01:00
- name: number_nights
data_type: integer
description: "number of nights for the booking"
2024-12-02 08:44:24 +01:00
- name: total_fee_local
data_type: double precision
2024-12-03 10:37:05 +01:00
description: "total fee in local currency for the booking"
2024-12-02 08:44:24 +01:00
2024-11-29 09:59:35 +01:00
- name: email_flag
data_type: text
description: "screening result for email"
- name: phone_flag
data_type: text
description: "screening result for phone"
- name: watch_list
data_type: text
description: "screening result of the guest"
- name: channel
data_type: text
description: ""
- name: checkin_at_utc
data_type: timestamp without time zone
description: "Timestamp of checkin for the booking"
- name: checkin_date_utc
data_type: date
description: "Date of checkin for the booking"
- name: checkout_at_utc
data_type: timestamp without time zone
description: "Timestamp of checkout for the booking"
- name: checkout_date_utc
data_type: date
description: "Date of checkout for the booking"
- name: is_cancelled
data_type: boolean
description: ""
- name: cancelled_at_utc
data_type: timestamp without time zone
description: "Timestamp of cancellation of the booking"
- name: cancelled_date_utc
data_type: date
description: "Date of cancellation for the booking"
- name: user_email
data_type: text
description: ""
- name: guest_email
data_type: text
description: ""
- name: guest_last_name
data_type: text
description: ""
- name: guest_first_name
data_type: text
description: ""
- name: guest_telephone
data_type: text
description: ""
- name: company_name
data_type: text
description: ""
- name: property_manager_name
data_type: text
description: ""
- name: property_manager_email
data_type: text
description: ""
- name: listing_name
data_type: text
description: ""
2024-12-03 10:37:05 +01:00
- name: listing_address
data_type: text
description: ""
2024-11-29 09:59:35 +01:00
- name: listing_town
data_type: text
description: ""
- name: listing_country
data_type: text
description: ""
- name: listing_postcode
data_type: text
description: ""
- name: pets_allowed
data_type: boolean
description: ""
- name: level_of_protection_amount
data_type: integer
description: ""
- name: level_of_protection_currency
data_type: text
description: ""
- name: status_updated_at_utc
data_type: timestamp without time zone
description: "Timestamp when status was last updated"
- name: status_updated_date_utc
data_type: date
description: "Date of last status update of the verification"
- name: updated_at_utc
data_type: timestamp without time zone
description: "Timestamp of last updated of the verification"
- name: updated_date_utc
data_type: date
description: "Date of last update of the verification"
- name: athena_creation_at_utc
data_type: timestamp without time zone
description:
"Athena timestamp referring to when the booking was created.
It's provided by Guesty, but is not mandatory.
In case of doubt use created_at_utc or created_date_utc fields"
- name: athena_creation_date_utc
data_type: date
description: "Athena date referring to when the booking was created.
It's provided by Guesty, but is not mandatory.
In case of doubt use created_at_utc or created_date_utc fields"
- name: created_at_utc
data_type: timestamp without time zone
description: "Timestamp of creation of the verification in the system"
- name: created_date_utc
data_type: date
description: "Date of creation of the verification in the system"
- name: int_monthly_aggregated_metrics_history_by_deal_by_time_window
description: |
This model aggregates monthly historic metrics for deals over different time windows.
It provides insights into bookings, listings, revenue, retained revenue and
additional metrics.
The data is segmented by deal and time window for detailed analysis.
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- date
- id_deal
- time_window
columns:
- name: date
data_type: date
description: |
The last day of the month or yesterday for historic metrics.
It's the same date as for KPIs related models.
tests:
- not_null
- name: id_deal
data_type: character varying
description: Id of the deal associated to the host.
tests:
- not_null
- name: time_window
data_type: character varying
description: |
Identifier of the time window used for the aggregation of the metrics.
tests:
- not_null
- accepted_values:
values:
- All History
- Previous 12 months
- Previous 6 months
- Previous 3 months
- Previous month
- name: metric_from_date
data_type: date
description: |
The first day of the month corresponding to the lower bound
range in which the metric is computed. It can be null if
there's no previous history for that deal. It can vary from
deal to deal depending on the number of months the deal has
been active.
- name: metric_to_date
data_type: date
description: |
The first day of the month corresponding to the upper bound
range in which the metric is computed. It can be null if
there's no previous history for that deal.
- name: main_deal_name
data_type: string
description: |
Main name for this ID deal.
tests:
- not_null
2025-01-09 16:31:22 +01:00
- name: has_active_pms
data_type: boolean
description: |
Does the deal have an active associated PMS.
tests:
- not_null
2025-01-09 16:31:22 +01:00
- name: active_pms_list
data_type: string
description: |
Name of the active PMS associated with the deal. It can have more than
2025-01-09 16:31:22 +01:00
one PMS associated with it. It can be null if it doesn't have any PMS associated.
- name: main_billing_country_iso_3_per_deal
data_type: string
description: |
ISO 3166-1 alpha-3 main country code in which the Deal is billed.
In some cases it's null.
- name: deal_lifecycle_state
data_type: string
description: |
Lifecycle state of the deal.
- name: deal_hubspot_stage
data_type: string
description: |
Hubspot stage of the deal.
In some cases it's null.
- name: account_manager
data_type: string
description: |
Account manager of the deal.
In some cases it's null.
- name: live_date_utc
data_type: date
description: |
Date when the deal went live according to
Hubspot. In some cases it's null.
- name: cancellation_date_utc
data_type: date
description: |
Date when the deal was cancelled according to
Hubspot. It can be null if the deal has never
churned.
- name: created_bookings
data_type: integer
description: |
Total amount of bookings created by the deal
in the time window. It can be null if no bookings
were created.
tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: listings_booked_in_month
data_type: decimal
description: |
Average amount of listings booked in month by the deal
in the time window. It can be null if no listings
were booked.
tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: total_revenue_in_gbp
data_type: decimal
description: |
Total revenue in GBP generated by the deal in the
time window. It can be null if no revenue was generated.
It can be negative.
- name: revenue_retained_in_gbp
data_type: decimal
description: |
Total revenue in GBP retained by the deal in the
time window, post host takeaway waivers.
It can be null if no revenue was retained.
It can be negative.
- name: waiver_paid_back_to_host_in_gbp
data_type: decimal
description: |
Total amount of waivers paid back to the host in GBP
in the time window. It can be null if no waivers were
paid back. It's displayed as a negative value.
- name: invoiced_revenue_in_gbp
data_type: decimal
description: |
Total amount of revenue in GBP invoiced to the host
in the time window. It considers both Operator revenue as
well as APIs revenue. It can be null if no revenue was
invoiced to the host. It can be negative.
- name: guest_payments_in_gbp
data_type: decimal
description: |
Total amount of payments in GBP made by the guest
in the time window. It can be null if no payments
were made by the guest. It can be negative.
tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: guest_revenue_retained_in_gbp
data_type: decimal
description: |
Total amount of revenue in GBP retained by the deal
from the guest in the time window, post host takeaway waivers.
It can be null if no revenue was retained from the guest.
It can be negative.
- name: host_resolution_payment_count
data_type: integer
description: |
Total amount of resolution payments made to the host
in the time window. It can be null if no resolution
payments were made by the host.
tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: host_resolution_amount_paid_in_gbp
data_type: decimal
description: |
Total amount of resolution payments made to the host
in GBP in the time window. It can be null if no resolution
payments were made by the host. It can be negative.
It's displayed as a negative value. In some extreme
cases, it can be higher than 0.
- name: revenue_retained_post_resolutions_in_gbp
data_type: decimal
description: |
Total amount of revenue in GBP retained by the deal
post waiver payouts and resolution payouts in the time window.
It can be null if no revenue was retained post resolution payments.
It can be negative, thus indicating that we are losing money.
2025-01-03 11:29:58 +01:00
- name: int_deals_consolidation
description: |
"This table contains all deal ids from different sources used in Superhog.
It contains the source (Hubspot, Xero or Core), the id_deal and the name"
columns:
- name: id_deal
data_type: character varying
description: "Unique ID for this deal."
tests:
- unique
- not_null
- name: core_company_name
data_type: character varying
description: "Company name of the deal as shown in Core."
- name: core_company_name_count
data_type: integer
description: "Count of distinct names the deal has in Core.
It might be the case that a deal has ony NULL value for a name,
so the count will be 0"
tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: hubspot_deal_name
data_type: character varying
description: "Name of the deal as shown in Hubspot."
- name: hubspot_deal_name_count
data_type: integer
description: "Count of distinct names the deal has in Hubspot.
It might be the case that a deal has ony NULL value for a name,
so the count will be 0"
tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: xero_contact_name
data_type: character varying
description: "Contact name of the deal as shown in Xero."
- name: xero_contact_name_count
data_type: integer
description: "Count of distinct names the deal has in Xero.
It might be the case that a deal has ony NULL value for a name,
so the count will be 0"
tests:
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: false
- name: is_deal_in_core
data_type: boolean
description: "Flag to indicate if the deal is in Core."
- name: is_deal_in_hubspot
data_type: boolean
description: "Flag to indicate if the deal is in Hubspot."
- name: is_deal_in_xero
data_type: boolean
description: "Flag to indicate if the deal is in Xero."