# Description
Exposes New KPIs of Bookings, Guest Journeys and Guest Payments in the 2 main flows: Monthly+MTD per Global, By Billing Country and By # of Listings Segmentation & Monthly By Deal
I did some checks on Global and By Billing Country and looks good.
I checked some examples for Monthly By Deal and looks good.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [NO] I have checked for DRY opportunities with other models and docs. **Likely there's ways to create default configs for KPIs saying "hey, this is Main KPIs for a specific view so use this models and these filters". But at this stage since it's a transitional stage I don't want do overthink. Let's tackle this later on if that's ok for you. It means we will live with some repeated expressions**
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #23455
# Description
Sets up warning for KPIs models that will be deprecated, which are:
* 13 in core
* 2 in xero
* 5 in cross
I will keep alive the cross models that handle the final aggregations for Main KPIs for the time being, as well as the newly created Churn model that has a dependency on the monthly by deal to be filled into the mtd flow. I think handling exposure logic for Main KPIs could be a separated migration.
In other words, this is already quite a bit to migrate.
# Checklist
**I just checked that dbt compiles correctly**
- [NA] The edited models and dependants run properly with production data.
- [NA] The edited models are sufficiently documented.
- [NA] The edited models contain PK tests, and I've ran and passed them.
- [NA] I have checked for DRY opportunities with other models and docs.
- [NA] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #23268
# Description
Simplifies the code for the models:
* int_monthly_12m_window_contribution_by_deal
* int_monthly_churn_metrics
By just removing the additive contribution approach. This also reduces the schema file information of these 2 models. I also adapted the description to clarify the state of the models.
No rush to merge this.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #22691
# Description
Re-aligned namings with Matt and Alex. This PR just changes the top losers, losers, winners and top winners to major decline, decline, gain and major gain respectively
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [NA] I have checked for DRY opportunities with other models and docs.
- [NA] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Rename of categories for top losers/winners
Related work items: #23170
# Description
Exposes Churn Rates into Main KPIs, specifically Revenue Churn Rate, Bookings Churn Rate and Listings Churn Rate. This is based on the average approach.
Additionally, it adds these 3 metrics in the kpis_additive_metrics_per_dimension_are_consistent test.
Additionally, it removes from int_mtd_vs_previous_year_metrics the computation of the additive Churn Rates. The removal of further unused code will be handled in a separated PR.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [NA] I have checked for DRY opportunities with other models and docs.
- [NA] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #22691
# Description
Changes:
* Explicit selection of fields in the last part of the query, rather than select *.
* Adding a few more Hubspot attributes, namely: AM, Hubspot Stage, Live Date and Cancellation Date. The main idea is to enrich the reporting with these.
* Adding the listings over 12 months. Here it's a bit more tricky than for Revenue or Bookings, since to me the main indicator is the amount of listings that are being booked in a month, over a period of 12 months (rather than unique count of listings that have been booked in the past 12 months). However, doing a sum of the listings booked in month will be very tricky for AMs and other users. I opted for averaging, and can be considered as, in average, a certain account has X amount of listings with bookings created, and this average considers the past 12 months. So I'd say it's a good estimate of the "real" size of a client in terms of potential for Superhog.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [NA] I have checked for DRY opportunities with other models and docs.
- [NA] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #22883
# Description
Small PR to ensure all values within the computation will be treated as decimals. I apply this at the beginning of the code for the 4 main inputs used. This avoids later integer divisions.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [NA] I have checked for DRY opportunities with other models and docs.
- [NA] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #22691
# Description
Adapts deals lifecycle logic by including offboardings from hubspot. It mostly increases the number of churning and inactive states in decrement of active state.
I also updated documentation and added an accepted values test.
When deploying and refreshing prod, figures in main kpis will be impacted
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [NA] I have checked for DRY opportunities with other models and docs.
- [NA] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #22689
# Description
Moves from intermediate/core to intermediate/cross the following models:
- `int_core__mtd_deal_lifecycle`
- `int_core__mtd_deal_metrics`
to their equivalents:
- `int_mtd_deal_lifecycle`
- `int_mtd_deal_metrics`
This also changes the schema entries, from core to cross, including changing the name of the model in the entry.
This also changes the dependencies, namely in `int_mtd_deal_metrics`, `int_mtd_vs_previous_year_metrics` and `int_monthly_aggregated_metrics_history_by_deal`.
This does NOT aim to alter the logic of the lifecycle in any case; it will be done in a separated PR.
Runs correctly end-to-end. We might need to drop the old models from production manually.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #22689
# Description
- Bugfix on nullif then 0 - it was applied to the numerator of Revenue computation, which made MoM growth be considered as null and propagated as 0 in the scores, which is not true.
- Bugfix on cast as numeric - this was introduced because PBI didn't read well some decimal figures when loading the data. However this impacted somehow in the score by some weird magic I don't understand. I just replace the casts by rounds, that are applied after the computation of the scores, and PBI seems happy with it.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [NA] I have checked for DRY opportunities with other models and docs.
- [NA] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #22635
# Description
Main changes:
* Includes 4 new fields to take into account 12 month created bookings. Specifically:
`deal_created_bookings_12_months_window`
`global_created_bookings_12_months_window`
`deal_contribution_share_to_global_created_bookings`
`deal_contribution_rank_to_global_created_bookings`
This also renames a CTE, that was previously stating it was revenue. Same for inline comments. Also includes documentation of this fields.
* Score range modification: Now, growth scores are multiplied by 100 and weighted score by 1000. This makes it easier to display and understand (Growth cannot be less than -100, threshold value is now -1, 0 and 1).
I checked that the content already in production has not change (ex: we still have the same 15 top losers for September).
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [NA] I have checked for DRY opportunities with other models and docs.
- [NA] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #22635
# Description
Main changes:
- Creation of `int_monthly_churn_metrics` model. This follows a similar approach as for mtd models, with jinja loops to aggregate the metrics at different dimensions. This reads from the previous monthly model, thus creating a dependency on Global KPIs with By Deal KPIs.
- Adds the 6 metrics in the main aggregated model of Global KPIs `int_mtd_vs_previous_year_metrics`. It doesn't necessarily mean that the 6 metrics will be made available in the report.
This does NOT display these metrics in the report, and won't be done until deal lifecycle is enriched to consider hubspot offboarding in the state "05-Churning".
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #22691
# Description
This PR creates a new model that depends on int_monthly_aggregated_metrics_history_by_deal. The idea is that this is used for Churn computation (Booking Churn, Revenue Churn, Listing Churn) later on.
The idea is relatively simple. Measure how much a Deal has been contributing to a Global amount (sum of metric for all deals) over the preceding period of 12 months. You will notice that there's 2 computations, the "additive" and the "average" one. This is because we still need to align with Matt/Suzannah on which approach makes more sense, but we need data for it. I'm not sure the namings are good though so happy to see your suggestions.
You will also notice that there's no filter by deal_lifecycle_state = '06-Churning'. This will be done in a separated model, whenever we attribute this model to the mtd computation. The reason is simple - this model stays at deal level, thus meaning we can do the dimension aggregation and even a lifecycle aggregation if needed, depending on the needs.
Be aware that this effectively means that MTD KPIs models will depend on the "monthly by deal" models. This has some cons in terms of dependency management but cannot be overcome since we the metric total revenue depends on many subsets. In essence, I don't see another way of doing it unless doing a massive KPIs refactor. I prefer to wait until the Product KPIs discussions are finished and then we see how we approach it.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #22691
# Description
Copies intermediate to reporting for growth score by deal. Schema is copy-paste from intermediate changing the model's name.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [NA] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #22635
# Description
Creates a model to identify deal growth based on YoY performance of Created Bookings, YoY performance of Listings Booked in Month and one month shifted YoY performance of Revenue.
Also added weighted score to account for revenue size.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs. **Probably something can be done here, sorry I've not checked.**
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #22635
# Description
Main changes:
* Guest revenue is now guest payments. PBI uses Guest revenue, so alias is changed at reporting level, while it uses guest_payments_in_gbp field.
* Removal of Waiver Amount Paid back to Host to Guest revenue and Total revenue.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [NA] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [NA] I have checked for DRY opportunities with other models and docs.
- [NA] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #22688
# Description
Adapts revenue figures in Main KPIs - MTD scope or global view. This includes MTD, Monthly Overview, Global Evolution over Time, Detail by Category. In essence, everything that is not by deal.
The changes are mainly 2:
* Remove the line that deducts the `Waiver Amount Paid Back to Hosts` in all metrics except the `Waiver Net Fees`. This effectively means that the previous `Guest Revenue` = `Guest Payments`, thus I dropped all 3 `Guest Payments` metrics.
* Do a renaming at metric display level, but not in the code. This means that I remove the computation of `guest_revenue_in_gbp` for instance and keep `guest_payments_in_gbp`, and apply the renaming later on, since the modelisation already accounts for defining metric names differently from those of the fields. For the rest of metrics, I revised all metrics name and did changes based on the [whiteboard](https://whiteboard.office.com/me/whiteboards/p/c3BvOmh0dHBzOi8vZ3VhcmRob2ctbXkuc2hhcmVwb2ludC5jb20vcGVyc29uYWwvcGFibG9fbWFydGluX3N1cGVyaG9nX2NvbQ%3d%3d/b!T2D3opQuBECSDnhuFZrUacFu3TxvSvdIsnI4Dxsh2IuaB1AigbciRqkqte61I4wz/01H5SI4J4L7HTPJGUT7JGYKTOSQYYWACXU). I also changed the dedicated data tests in Main KPIs to ensure it's working. I also changed the exclusion logic in reporting based on the name of the metric to not display metrics that depend on the invoicing cycle unless it's 2 months ago or before.
To keep in mind:
* Merging this will automatically display the new figures/naming in production. Might be wise to communicate to stakeholders since some key metrics (namely, Guest Revenue / Total Revenue) will change the meaning.
* We also need to do these changes in the metrics by deal part of the computation. I'd do first the removal of these fields in the PBI report (and take the opportunity to change the Data Catalogue) and then do the PR in DWH to change the logic. Before that though let's check that the names included in this PR are the correct ones :)
# Checklist
- [X] The edited models and dependants run properly with production data.
- [NA] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [NA] I have checked for DRY opportunities with other models and docs.
- [NA] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #22688
# Description
Changes (only in intermediate):
* Applies sqlfmt in KPIs source models (for some of them it was already applied). Specifically, the 3 Core models ONLY contains formatting changes
* Adds `main_deal_name` and `main_billing_country_iso_3_per_deal` in `int_monthly_aggregated_metrics_history_by_deal`
* Adds the 2 new fields in the schema entry of `int_monthly_aggregated_metrics_history_by_deal`, including the dbt test not null in the deal name.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #18911, #19083
# Description
Changes:
* Adding `main_deal_name` and `main_billing_country_iso_3_per_deal` in `int_dates_by_deal` model.
* Documents the 2 new fields. Also, ensures `main_deal_name` is not null
* Removes `id_deal not null` condition since it's enforced on the inner join with `int_core__deal`
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #18911, #19083
# Description
Before deploying KPIs by Billing Country, we spotted some issues that were basically increases on the volumes of any metric on the by billing country dimension that was based on Deal. This means, `int_core__mtd_deal_metrics` and `int_xero__mtd_invoicing_metrics`.
This PR changes the following:
* Now the 2 abovementioned models depend on the `int_core__deal` model, instead of `int_core__user_host` (thus removing duplicated stuff)
* Now all models use the main billing country at deal level, instead of doing it so at host level. The reason is that some small amount of hosts that share the same deal can have a different billing country. To avoid weird stuff, everything points to this simplification - that in general, it's not a massive change in the output.
* In order to do so easily, the 3 main billing country per deal fields have been propagated to `int_core__user_host`
To exemplify the solution, find here a snapshot of the differences in behavior:
```
select
dimension,
sum(deals_booked_in_month) as deals_booked_1,
sum(deals_booked_in_6_months) as deals_booked_6,
sum(deals_booked_in_12_months) as deals_booked_12,
sum(total_revenue_in_gbp) as total_revenue,
sum(xero_operator_net_fees_in_gbp) as operator_revenue,
sum(xero_booking_net_fees_in_gbp) as booking_fees,
sum(xero_listing_net_fees_in_gbp) as listing_fees,
sum(xero_verification_net_fees_in_gbp) as verification_fees,
sum(total_guest_revenue_in_gbp) as guest_revenue,
sum(xero_waiver_paid_back_to_host_in_gbp) as waiver_paid_back_to_hosts,
sum(waiver_net_fees_in_gbp) as waiver_net_fees
from intermediate.int_mtd_vs_previous_year_metrics
where date in ('2024-01-31')
group by 1
order by 1
```
Production:
vs.
Local:
Keep in mind that still Global dimension can be greater than any other dimension aggregated since not all users have a deal. Mismatches between the other 2 dimensions might be linked to the dump.
Commits are meaningful and help navigate in the changes.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #20823
# Description
Adds Billing Country dimension in KPIs, but does not expose them to reporting yet.
Silly thing, based on the macros I built, I cannot make incremental changes unless changing all models. This will need to be adapted, happy to hear your thoughts on how we do it.
Additionally, I have lack of performance of the model `mtd_guest_payments_metrics`. It takes around 5 min to execute, but technically the end-to-end runs in one shoot without breaking.
It's a complex PR because it changes many files, but you will see that:
* It mostly changes the join conditions for the dimensions or the schema tests,
* I tried to be very careful and add things step-by-step in the commits.
Goal is NOT to complete the PR yet until we see how we can improve performance. I can say though that data end-to-end looks ok to me, but would benefit from checking with production data for the new dimension
Update 30th Aug
* Added a new commit that includes `id_user_host` in `int_core__verification_payments`. Happy to discuss if it makes sense or not. But it changes the execution from ~600 sec to ~6 sec because it avoids a massive repeated join with `verification_requests`.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models. **To check because of performance issues**
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19082
# Description
Exposes Billable Bookings metric for KPIs, both in the "global+dimension" view and in the "deal" view.
Metrics have already been created for a while. Exposing them now after the changes carried out in the model `int_core__booking_charge_events`. Based on the current quality of the data, I opted for "Est. Billable Bookings" to account for the fact that this is an estimation. If you don't feel comfortable with it, let's remove the "Est. ".
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #18111
# Description
Fixes int_dates_by_deal tests
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #20318, #20319
# Description
This PR ensures the propagation of the dimensions for KPIs across the key aggregating and exposing models. Additionally, provides these 2 new fields in reporting while **not affecting the current data display**, thus it's safe to work in the PBI report without needing to work in 2 PRs in parallel.
**Changes:**
**1 - Intermediate, `int_mtd_vs_previous_year_metrics`:**
* Removes the temporary filter on `where dimension in ({{ production_dimensions }})`. This will be applied directly to reporting later. This ensures that the new dimension on customer segmentation is fully available only within intermediate.
* Adds `dimension` and `dimension_value` granularity. This includes: 1) adding these fields, 2) joining by these fields with all the source CTEs containing the source models with metrics - which in turn needs the change of the dates model - and 3) joining by these fields in the self-join to compute the incremental vs. previous year.
* Changes on the schema file
**2 - Intermediate, `int_mtd_aggregated_metrics`:**
* Adds `dimension` and `dimension_value` granularity. This includes only adding these fields.
* Changes on the schema file
**3 - Reporting, `mtd_aggregated_metrics`:**
* Adds the filter removed on `int_mtd_vs_previous_year_metrics`. This ensures that only the Global dimension is available for the reporting, thus **no changes from user POV**.
* Adds `dimension` and `dimension_value` granularity. This includes only adding these fields
* Changes on the schema file
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19325
# Description
Silly change:
* Modifies `int_core__mtd_guest_payments_metrics` to apply the proper key on date, dimension and dimension_value.
* -> *The weird thing is that the previous dbt test I run worked well. Is it possible that the configuration in the schema file prevails on top of the model configuration? I thought it was the other way around...*
Main changes:
* Modifies `int_xero__mtd_invoicing_metrics` to include the customer segmentation based on listings.
* `schema.yaml` is also affected including new fields and tests
* Added the macro to retrieve the production dimension in `int_core__mtd_vs_previous_year_metrics` to avoid propagating this upwards and messing up with the data display.
Overall, follows a similar strategy as we did for Booking, Guest Journey, Deal, Accommodation and Guest Payments metrics. For reference, [here's the previous PR on Guest Payments](https://guardhog.visualstudio.com/Data/_git/data-dwh-dbt-project/pullrequest/2580).
This is the last PR on the source models for KPIs. Will follow: propagation + exposure
# Checklist
- [X] The edited models and dependants run properly with production data.
* **Important note**: this segmentation provides null values for all API-related KPIs. Makes sense, since the 4 deal id we have for APIs do NOT have, or have had, a listing linked to them. I'd say it's not a blocker.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19325
# Description
Modifies `int_core__mtd_guest_payments_metrics` to include the customer segmentation based on listings. `schema.yaml` is also affected including new fields, tests and apply the proper naming (from guest revenue to guest payments). I also modified a silly naming that was referring to deals to refer to listings/accommodations, my bad.
Added the macro to retrieve the production dimension in `int_core__mtd_vs_previous_year_metrics` to avoid propagating this upwards and messing up with the data display.
Overall, follows a similar strategy as we did for Booking, Guest Journey, Deal and Accommodation metrics. For reference, [here's the previous PR on Accommodations](https://guardhog.visualstudio.com/Data/_git/data-dwh-dbt-project/pullrequest/2575?_a=overview).
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19325
# Description
Takes into account @<Pablo Martín> 's feedback from the previous PR, slightly modified. This PR separates 1) the dimensions while developing vs. 2) the dimensions once these are available for production. This are within the same file of macro configuration for KPIs, namely `business_kpis_configuration`.
End-goal, all CTEs in `int_mtd_vs_previous_year_metrics` will read from this new macro `get_kpi_dimensions_for_production`, so eventually we won't need any hardcode once we want to add new dimensions. In the meantime, I'll be adding this new line for each PR (still 2 missing :D)
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [ ] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19325
# Description
Modifies `int_core__mtd_accommodation_metrics` to include the customer segmentation based on listings. `schema.yaml` is also affected including new fields and tests. Hardcoded `int_core__mtd_vs_previous_year_metrics` to avoid propagating this upwards and messing up with the data display.
Overall, follows a similar strategy as we did for Booking, Guest Journey and Deal metrics. For reference, here's [the previous PR on Deal](https://guardhog.visualstudio.com/Data/_git/data-dwh-dbt-project/pullrequest/2534). I noticed that I mixed the schema tests of Deals and Accommodations, this PR should fix both.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19325
# Description
Modifies `int_core__mtd_deal_metrics` to include the customer segmentation based on listings. `schema.yaml` is also affected including new fields and tests. Hardcoded `int_core__mtd_vs_previous_year_metrics` to avoid propagating this upwards and messing up with the data display.
Overall, follows a similar strategy as we did for Booking and Guest Journey metrics. For reference, [here's the previous PR on GJ](https://guardhog.visualstudio.com/Data/_git/data-dwh-dbt-project/pullrequest/2533).
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19325
# Description
Modifies `int_core__mtd_guest_journey_metrics` to include the customer segmentation based on listings. `schema.yaml` is also affected including new fields and tests. Hardcoded `int_core__mtd_vs_previous_year_metrics` to avoid propagating this upwards and messing up with the data display.
Overall, follows a similar strategy as we did for Booking metrics.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19325
# Description
_Describe your motivation and changes here._
# Checklist
- [ ] The edited models and dependants run properly with production data.
- [ ] The edited models are sufficiently documented.
- [ ] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
# Description
Added relative_increment_with_sign_format for special formatting in PBI
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Added relative_increment_with_sign_format
Reverts !2524
Related work items: #19559
# Description
What's new:
- Creation of `get_kpi_dimensions`: new macro to have a single point of source of configuration for dimensions for the KPIs. It's a way to enforce global variables on-demand. I kind of like this approach and we could do it for Xero models as well :)
- Modification of `int_core__mtd_booking_metrics` and `int_dates_mtd_by_dimension`: removal of duplicated code within the dimension context. Uses Jinja code and applies different configurations depending on the dimension chosen. Still, different metrics are placed in different CTEs. I believe it might be possible to also configure metrics BUT at the cost of over-complexifying the macro logic, so I wouldn't go for it at this stage.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] **I have checked for DRY opportunities with other models and docs.**
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19325
# Description
This is a first idea of how I'd like to add dimensionality in the KPIs for the mtd models. For the moment, I keep deal_id apart, so I just touch the "mtd" models, that so far only contained "global" metrics.
In this case I include the listing segmentation (0, 1-5, 6-20, etc) in the bookings. To do this, I created 2 new fields: dimension and dimension_values.
I also created a "master" table with `date` - `dimension` - `dimension_value` called `int_dates_mtd_by_dimension`
Important notes:
- I force a hardcode in `int_mtd_vs_previous_year_metrics`. This is to not break production.
- You will notice how repetitive the code is starting to look. My intention with this PR is that we are happy with this approach on the naming, the strategy for joins, etc. If that's ok, next step is going to be doing macros on top. Think of the state of `int_core__mtd_booking_metrics` as the "compiled version" of the macro that should come afterwards.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19325
# Description
Materialise int_dates_mtd and int_dates_by_deal as tables. This should improve the run speed as seen in local by quite a bit, and hopefully provide a better starting point for adding new dimensionality on business kpis.
I also documented these 2 models, that were missing :)
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [] Check if a full-refresh is required after this PR is merged.
Related work items: #19514
After discussion with Pablo on the fact that Deposits are only with status "Paid" for a given time before they get Cancelled or Refunded, we just believe it's best to remove the Deposits amount from the Guest Payments metric. In any case, this does not represent a Revenue source... This was discovered while doing the data quality assessment for revenue figures ([here](https://www.notion.so/knowyourguest-superhog/Data-quality-assessment-DWH-vs-Finance-revenue-figures-6e3d6b75cdd4463687de899da8aab6fb))
Before, `total_guest_payments_in_gbp` was a standalone metric that computed any payment by the guest with status paid. We were computing revenue based on the `total_guest_income_in_gbp`, which mainly was the sum of waiver payments, deposit fees (not deposit itself!) and check in hero fees.
Mainly what I did is:
- remove the existing `total_guest_payments_in_gbp` in the source models (int_core__xxx_guest_payments_xxx)
- rename the already existing `total_guest_income_in_gbp` to `total_guest_payments_in_gbp`
Related work items: #18787, #18914
Exposes the following metrics:
1. Invoiced Booking Fees
1. Invoiced Listing Fees
1. Invoiced Verification Fees
1. Invoiced Guesty Fees
1. Invoiced E-Deposit Fees
1. Deposit Fees
1. Waiver Amount Paid by Guests
1. Waiver Amount Paid back to Hosts
1. Check-In Hero Amount Paid by Guests
This PR is only for Global metrics, the one by deal id will follow separately.
Keep in mind that merging this PR will make the data appear in the report.
Related work items: #18914
Small refactor to follow up on last week's PR. We removed from the Guest Revenue models the host-takes-waiver aspect, thus these models are now only depending from Core. We just need to migrate it from cross to core.
One small detail as well, since we do not take into account at these models level the host-takes-waiver, technically, I would not call these models revenue but rather Guest Payments. This is why I also took the opportunity to apply this name.
Changes:
- `int_monthly_guest_revenue_by_deal` is now `int_core__monthly_guest_payments_history_by_deal`, and the location has changed from `intermediate.cross` to `intermediate.core`
- `int_mtd_guest_revenue_metrics` is now `int_core__mtd_guest_payments_metrics`, and the location has changed from `intermediate.cross` to `intermediate.core`
- Schema changes, moving these 2 models' documentation with the new naming from Cross to Core
- Provide continuity in following dependants: `int_mtd_vs_previous_year_metrics` and `int_monthly_aggregated_metrics_history_by_deal` now read from the 2 new models respectively. Additionally, the model alias has changed from `guest_revenue` to `guest_payments` to keep consistency.
This PR does not expose new metrics, but should keep the existing ones unaffected.
Related work items: #18914
Adding submetrics of guest revenue by deal:
- deposit_fees_in_gbp
- checkin_cover_fees_in_gbp
- waiver_payments_in_gbp
all of this adds up to
- total_guest_income_in_gbp
and the total_guest_revenue_in_gbp is now computed by subtracting waivers paid to hosts, coming from the invoiced model. This also affects the total revenue computation and the weighted metrics.
This PR it's completely equivalent to the [Adding submetrics of guest revenue](https://guardhog.visualstudio.com/Data/_git/data-dwh-dbt-project/pullrequest/2381?path=/models/intermediate/cross/int_mtd_guest_revenue_metrics.sql&_a=files), that was already merged for the global view. The only difference is that this one is for the deal-based part. It does not expose the metrics to the report.
A follow-up PR is expected to apply the correct naming conventions for guest revenue models.
I recommend checking the first commit, the one that applies the changes, and should be easily understandable with the link to the previous PR. There's a couple of additional commits that only affect formatting.
Related work items: #18787, #18914
Adding submetrics of guest revenue:
- deposit fees
- checkin cover fees
- waiver payments
all of this adds up to
- guest income
and the revenue is computed by subtracting waivers paid to hosts
Related work items: #18787
This PR exposes the following metrics to the Main KPIs business overview report, for both Global + By Deal view:
- Total Revenue
- Total Revenue per Booking Created
- Total Revenue per Guest Journey Created
- Total Revenue per Deals Booked in Month (does not apply on the by deal view)
- Total Revenue per Listings Booked in Month
- Invoiced Operator Revenue
- Host Resolution Payment Count
- Host Resolution Amount Paid
Keep in mind Global view will be displaying these metrics once this is merged. I also changed a bit the order of the metric display.
Note that Billable Bookings are not included.
I recommend to review by 1) checking the first commit. This is almost the same as the previous abandoned PR that @<Joaquin Ossa> you already checked on Tuesday. I added a second commit, to be checked later, which basically fixes some stupid issues that if one of the source of revenue is null, then total revenue is null. This is specially critical for the view by deal, since most of them do not have revenue from APIs - thus all total revenue figures were null...
Related work items: #18108, #18109, #18110, #18719
Computing and propagating APIs revenue metrics.
I retrieved the revenues linked to Guesty and e-deposits. The sum of those are considered the total API revenue at this stage.
These 3 metrics are available in upper layers (not exposed yet to the report), just to fix the total revenue computation, which now includes APIs revenue
Related work items: #18719
Propagates billable bookings kpis in intermediate. Does not expose any metric to the report.
Changes:
- Retrieval and computation of previous_year and relative_increment for global view (mtd models)
- Retrieval as is for deal view
Related work items: #18111
This PR aims to propagate the invoicing metrics through the DWH. It does not expose them to users, yet.
This PR effectively computes the following metrics, for both the "global" view (MTD) and the "by deal" view (by_deal):
- Invoiced Operator Revenue
- Host Resolution Count of Payments
- Host Resolution Amount Paid
With these 3 new metrics, we're able to combine them with the existing ones to compute:
- Total Revenue
- Total Revenue per Booking Created
- Total Revenue per Guest Journey Created
- Total Revenue per Deal Booked in Month
- Total Revenue per Listings Booked in Month
You'll also note that I've included standalone metrics for booking fees, listing fees, verification fees and waiver payments. This will not be exposed in this batch 2, but based on the conversation with Finance, will clearly make it for batch 3. I just find it easier to add it now, since it's straight forward.
Main changes:
- `int_mtd_vs_previous_year_metrics` now computes all the above mentioned metrics
- `int_monthly_aggregated_metrics_history_by_deal` now computes all the above mentioned metrics, except Total Revenue per Deal Booked in Month since it does not make sense for the deal view. Additionally, I took the opportunity to include the missing metrics from listings (accommodations). The goal is not necessarily to display them, but at least compute it on our side.
Additional changes:
- In `int_xero__mtd_invoicing_metrics` and `int_xero__monthly_invoicing_history_by_deal`, there's a very silly name change to keep the same convention for fees: from `xero_operator_net_fees` to `xero_operator_net_fees_in_gbp`
- I applied additional changes in `int_monthly_aggregated_metrics_history_by_deal` with the goal to keep the same format as we have in `int_mtd_vs_previous_year_metrics`, this meaning:
1 - explicit alias naming (from `gj` to `guest_journeys`)
2 - keep a similar arrangement of metrics, and clearly separate scopes depending on the metric type
3 - Re-apply autoformatting
Related work items: #18108, #18109, #18110
