# Description
Remove old models:
- `monthly_aggregated_metrics_history_by_deal`
- `int_monthly_aggregated_metrics_history_by_deal`
# Checklist
- [ ] The edited models and dependants run properly with production data.
- [ ] The edited models are sufficiently documented.
- [ ] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Remove old models
Related work items: #29672
# Description
Updates deprecation dates to 23rd May.
Updates exposure with new Account Growth report
# Checklist
- [X] Project compiles
Related work items: #29374
# Description
Propagates the new account growth model to reporting.
Rounds and a couple of additional fields are handled here for PBI purposes.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #29374
# Description
Creates an intermediate model to compute the impact in revenue (total or rrpr) due to the growth.
This model also categorises the impact, and provides additional attributes for reporting purposes.
This model aims to substitute, in time, the current existing version of `int_monthly_growth_score_by_deal`.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #29374
# Description
New model for Resolutions payments
# Checklist
- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [x] I have checked for DRY opportunities with other models and docs.
- [x] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #29373
# Description
Removes exposures of monthly_aggregated_metrics_history_by_deal in Main KPIs
Project compiles.
# Checklist
N/A
Related work items: #29204
# Description
I opted to name the combination of (Platform) Billable Bookings and (API) Billable Verifications as Billable Items. This is to ensure consistent naming.
Changes:
* Renamed `int_kpis_projected__agg_daily_billable_bookings` to `int_kpis_projected__agg_daily_billable_items`. This now has a new CTE named combination_of_billable_items that combines both API and Platform billable items. Renamed any "bookings" field to "items".
* Renamed `int_kpis_projected__agg_monthly_billable_bookings` to `int_kpis_projected__agg_monthly_billable_items`. Renamed any "bookings" field to "items".
* Renamed `int_billable_bookings_growth_score_by_deal` to `int_billable_items_growth_score_by_deal`. This now has a new CTE named `aggregated_monthly_billable_items` that combines the historical information both API and Platform on billable items. Renamed any "bookings" field to "items".
* Changes Schema accordingly. Small note here that the assert dimension completeness stops working since there's dimensions specific to API or Platform, thus I removed it.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #29374
# Description
Changes:
* 4 new models in the scope of KPIs for Billable Verifications from APIs. I believe it's more correct to say these are Billable Verifications than Billable Bookings since there's some cases in which a Booking can be duplicated and it's billed multiple times. These include:
* A daily metric model - extremely simple. You will notice there's no Billing Country not Listing Segmentation. This is because for ALL API cases this is UNSET, thus, I just remove it.
* An equivalent monthly metric model.
* Two aggregated models per dimension, dimension value: on a daily and a monthly basis.
Important change: the macro that handles the aggregations sets by default Billing Country and Listing Segmentation. I modified a bit the flow so the only required dimension is Global, and these are skipped for APIs models.
This is needed for the changes intended on the Growth score. In there, I'll combine both Platform Billable Bookings with API Billable Verifications.
Notice there's no MTD models. These could be added for sure; but since I'm not creating any metric in Main KPIs or similar, I opted to skip it for now. It can be done later on.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #29374
# Description
Switches Created Bookings to Billable Bookings in the scope of Projected Bookings. I opted for hard switching (thus removing Projected Created Bookings) altogether to avoid having unused models in production.
This aims to set the ground to include, in the future, APIs Billable Bookings.
Impact in KPIs:
* `int_kpis__agg_daily_created_bookings` now is `int_kpis__agg_daily_billable_bookings`. Aside from the name change, it's reduced to 4 metrics to just one. Schema changes also apply.
Impact in Projected KPIs:
* `int_kpis_projected__agg_daily_created_bookings` now is `int_kpis_projected__agg_daily_billable_bookings`. Fields are also renamed accordingly.
* `int_kpis_projected__agg_monthly_created_bookings` now is `int_kpis_projected__agg_monthly_billable_bookings`. Fields are also renamed accordingly.
* Schema file is also updated accordingly.
Impact on new model for Growth Score:
* `int_created_bookings_growth_score_by_deal` now is `int_billable_bookings_growth_score_by_deal`. Fields are also renamed accordingly.
* Schema is also updated accordingly.
Note that since the end model `int_billable_bookings_growth_score_by_deal` is not used at the moment in reporting, there's no user-facing impact. Also, the rest of modified models are not used for other purposes.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #29374
# Description
Creates a first unified version of API verifications. This is a very simple model as it aims to capture the minimal essence of Verifications in a unified view. Any very-in-depth-API-specific detail is not available here.
Additionally, I propagated the Check-in date in the Athena model.
This is needed for API Bookings KPIs future models.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models. **Currently as a view, despite it's 70k records. If this grows we could consider other materialisations**
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #29374
# Description
Removes UNKNOWN in Service Business Type linked to 'BASIC DAMAGE DEPOSIT OR BASIC WAIVER'.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Set option with basic waiver as deposit management
Related work items: #29649
# Description
After checking the current growth score model... I think I will decommission it for something simpler.
This PR introduces a new growth score purely based on Created Bookings, thus it won't work for API deals. For API deals I will do a similar Total Revenue comparison separately.
This uses the projected bookings at the end of the month, thus this is updated in a daily basis and it's timely.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #29374
# Description
Small refactor on Growth Score model. It just handles the dimension deals attributes at the end instead of having these in each CTE.
I also forced a partition ordered by deal AFTER the current ordering. This is a side effect because SQL randomly choses a value if the value being used to order was exactly the same (ex: revenue = 0). This was causing issues when auditing the refactor.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #29374
# Description
Technically, a payout can be submitted while not in a finished incident state. Likely next step from res perspective is just to close the ticket.
This PR just adds two new fields for completion and consistency, and fixes the completion test so it makes sense.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
# Description
Added new invoices metrics for CIH and S&P API for KPI and Accounting models
# Checklist
- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
New API invoices
Related work items: #29276
# Description
Adds monetary value in terms of payouts for further understanding of the potential impact on improving the flagging.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #29284
# Description
Creates 2 new models in the scope of flagging: how good are we at identifying "at risk" bookings vs. 1) the number of claims generated and 2) the number of submitted payouts?
This only applies for Protected Bookings in New Dash that have been completed (14 days after the check-out) with potential resolutions appearing in Resolutions Center.
The first table `int_flagging_booking_categorisation` contains all the heavy logic to categorise the bookings.
The second view `int_flagging_performance_analysis` computes standard binary classification scores, for the 2 possible ways of tracking.
Tables are already in prod to help you understand while reviewing. You'll see that the figures are still quite low, specially due to small amount of claims/submitted payouts. This makes the true positives being just... 1.
There's heavy test and documentation coverage to ensure there's no mistakes on the computation.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models. **Materialising as table the first model despite being just 1 record since otherwise tests takes ages**
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #29284
# Description
Adds more robust testing for Total Revenue and Revenue Retained Post-Resolutions in:
* int_mtd_vs_previous_year_metrics
* int_ytd_mtd_main_metrics_overview
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
More robust testing on key metrics
Related work items: #29227
# Description
Now that we finally have customers on CIH API
Nathan noticed an error on the report which we hadn't noticed before, the fees were not included in the reporting model and was previously being taken from the monthly model which did have them. So currently the monthly value is being displayed correctly but not the individual fees for each verification.
# Checklist
- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
cih api fix
Related work items: #29225
# Description
As raised by Kayla, Retained metrics are not computed correctly in AM Account Margin report, see screenshot:
Issue lies in the fact that Guest Payments contains more dimensions than the rest of the models. This is generating duplicates in the combination_of_sources CTE, but these are effectively hidden because of the sum() applied of the metrics.
In the case of a Deal having in the same day Guest Payments with different dimensions from those selected, AND on the same day having a value on Invoiced Revenue or Host Resolutions, then the Invoiced Revenue or Host Resolutions would be duplicated.
This PR aims to fix this issue, and also adds a few tests on the base model for AM Account Margin to ensure this is kind of error raises an alert.
I'll be debugging other affected models and increase the robustness test-wise; as well as gathering the full impact.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #29227
# Description
Small changes to exclude "irrelevant" data. All data is relevant I guess but this is less.
* Exclude churned accounts after 3 months.
* Exclude any data after 24 months.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #28998
# Description
Added `has_upgraded_screening_services_business_type`
So in the report we can filter by this because currently all bookings should have `has_screening_services_business_type` with BASIC SCREENING
# Checklist
- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
upgraded screening
Related work items: #28640
# Description
Creates a new model for reporting. It includes any relevant metric at Deal level in a similar format as we do for `mtd_aggregated_metrics`. Additionally, there's few Deal attributes - from Hubspot, segmentations, lifecycles, etc.
In order to dynamically choose which metrics are relevant on a Deal level, I modified the configuration in `int_mtd_aggregated_metrics` so the extraction is under control.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models. **Worth discussing the possibilities to include indexes in the future**
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #28998
# Description
To be able to compute By Deal KPIs similarly as we do for Main KPIs we need to propagate the "By Deal" dimension first in intermediate. This does not impact reporting since there's a macro that already cuts the dimensions to be displayed, but I modified the name so it's clearer.
Changes:
* In `int_mtd_vs_previous_year_metrics`, I added a new `dimension_list`. This is applied to all initial CTEs that filter by dimension. Note that I added here the `by_deal` dimension.
* Modified the name of `get_kpi_dimensions_for_production` to `get_main_kpis_dimensions_for_production`. Now it's more explicit that this is the configuration for Main KPIs reporting. This affects both `business_kpis_configuration` and it's usage in `mtd_aggregated_metrics`.
* Modified the tests in `int_mtd_vs_previous_year_metrics` and `int_mtd_aggregated_metrics` to include the new dimension `by_deal`.
* It seems by adding this now autoformatting works again on this model! I'll tag all cases that are just because of autoformatting.
MD5 trick has been applied in `reporting.mtd_aggregated_metrics`.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #28998
# Description
Removes old onboarding mrr models, that are no longer used.
dbt compiles correctly and no deprecated warning is showing anymore.
This finishes the refactor
Related work items: #28949
# Description
Puts in production the new KPIs model for Onboarding MRR.
The change has been validated with MD5 trick.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #28949
# Description
Creates a single model in KPIs, aggregated by dimension/dimension value, in a monthly basis; named: `int_kpis__agg_monthly_onboarding_mrr`.
Attention! This substitutes both current models, namely `int_monthly_onboarding_mrr_per_deal` and `int_mtd_agg_onboarding_mrr_revenue`. However, both models are currently used to retrieve data for Main KPIs. I just combined both into one, so it will simplify also the gathering of data later on.
Note that this model is special in the sense that the onboarding mrr per deal is computed for global, listing segmentation and billing country dimensions; while the total onboarding mrr is only done for global and listing segmentation (as it's based on the number of listings segmentation for the total compute).
This has been tested with dbt audit helper and md5. This has been a bit more complex since it's comparing 1 new model vs 2 existing models. For reference, this is the md5 comparison used:
```
SELECT md5(array_agg(md5((t1.*)::varchar))::varchar)
FROM (
SELECT
mrr.date,
mrr.dimension,
mrr.dimension_value,
mrr.expected_mrr_per_deal,
total.number_of_new_deals,
total.expected_mrr as total_expected_mrr
FROM intermediate.int_monthly_onboarding_mrr_per_deal mrr
left join intermediate.int_mtd_agg_onboarding_mrr_revenue total
on mrr.date = total.date
and mrr.dimension = total.dimension
and mrr.dimension_value = total.dimension_value
ORDER BY date, dimension, dimension_value
) AS t1
union all
SELECT md5(array_agg(md5((t2.*)::varchar))::varchar)
FROM (
SELECT *
FROM intermediate.int_kpis__agg_monthly_onboarding_mrr
ORDER BY date, dimension, dimension_value
) AS t2
```
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #27120, #28949
# Description
Tags as deprecated the following models:
* `int_monthly_onboarding_mrr_per_deal`
* `int_mtd_agg_onboarding_mrr_revenue`
Project compiles.
Related work items: #28949
# Description
Removes the models:
* int_monthly_12m_window_contribution_by_deal
* int_monthly_churn_metrics
as well as their entries in the schema files.
Project compiles and KPIs run works.
This closes stage 3 of the refactor.
# Checklist
- [ ] The edited models and dependants run properly with production data.
- [ ] The edited models are sufficiently documented.
- [ ] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Remove previous Churn models
Related work items: #28948
# Description
Small but important change.
Churn metrics will be propagated in production by sourcing intermediate/kpis rather than the cross models.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #28948
# Description
Creates a new model in intermediate/kpis called `int_kpis__agg_monthly_churn_contribution`. This follows the convention (aggregated = by dimension and dimension value) and monthly (1 dimension value per dimension per date where date is unique in month).
This single model aims to substitute, in the future, the 2 churn models simultaneously:
* `int_monthly_12m_window_contribution_by_deal`
* `int_monthly_churn_metrics`
Main changes:
* There's no longer a dependency with `int_monthly_aggregated_metrics_history_by_deal`. Rather, it gets the metrics and dimensions from wherever needed and nothing else.
* I also took the opportunity to clean any code that was not needed after combining both models, resulting in dropping a whole CTE.
* Updated schema description for clarification purposes.
The output of the new model is identical to the 2nd model, `int_monthly_churn_metrics`, confirmed with the md5 trick. Model runs in 1.45 seconds in my laptop so no performance issue (it's technically faster than running the 2 current models which is around 6.3 seconds but whatever).
Note that this has NO effect yet on production. The switch will be handled in a separated PR.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #28948
