# Description
Switches Created Bookings to Billable Bookings in the scope of Projected Bookings. I opted for hard switching (thus removing Projected Created Bookings) altogether to avoid having unused models in production.
This aims to set the ground to include, in the future, APIs Billable Bookings.
Impact in KPIs:
* `int_kpis__agg_daily_created_bookings` now is `int_kpis__agg_daily_billable_bookings`. Aside from the name change, it's reduced to 4 metrics to just one. Schema changes also apply.
Impact in Projected KPIs:
* `int_kpis_projected__agg_daily_created_bookings` now is `int_kpis_projected__agg_daily_billable_bookings`. Fields are also renamed accordingly.
* `int_kpis_projected__agg_monthly_created_bookings` now is `int_kpis_projected__agg_monthly_billable_bookings`. Fields are also renamed accordingly.
* Schema file is also updated accordingly.
Impact on new model for Growth Score:
* `int_created_bookings_growth_score_by_deal` now is `int_billable_bookings_growth_score_by_deal`. Fields are also renamed accordingly.
* Schema is also updated accordingly.
Note that since the end model `int_billable_bookings_growth_score_by_deal` is not used at the moment in reporting, there's no user-facing impact. Also, the rest of modified models are not used for other purposes.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #29374
# Description
Creates a first unified version of API verifications. This is a very simple model as it aims to capture the minimal essence of Verifications in a unified view. Any very-in-depth-API-specific detail is not available here.
Additionally, I propagated the Check-in date in the Athena model.
This is needed for API Bookings KPIs future models.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models. **Currently as a view, despite it's 70k records. If this grows we could consider other materialisations**
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #29374
# Description
Removes UNKNOWN in Service Business Type linked to 'BASIC DAMAGE DEPOSIT OR BASIC WAIVER'.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Set option with basic waiver as deposit management
Related work items: #29649
# Description
After checking the current growth score model... I think I will decommission it for something simpler.
This PR introduces a new growth score purely based on Created Bookings, thus it won't work for API deals. For API deals I will do a similar Total Revenue comparison separately.
This uses the projected bookings at the end of the month, thus this is updated in a daily basis and it's timely.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #29374
# Description
Small refactor on Growth Score model. It just handles the dimension deals attributes at the end instead of having these in each CTE.
I also forced a partition ordered by deal AFTER the current ordering. This is a side effect because SQL randomly choses a value if the value being used to order was exactly the same (ex: revenue = 0). This was causing issues when auditing the refactor.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #29374
# Description
Technically, a payout can be submitted while not in a finished incident state. Likely next step from res perspective is just to close the ticket.
This PR just adds two new fields for completion and consistency, and fixes the completion test so it makes sense.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
# Description
Added new invoices metrics for CIH and S&P API for KPI and Accounting models
# Checklist
- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
New API invoices
Related work items: #29276
# Description
Adds monetary value in terms of payouts for further understanding of the potential impact on improving the flagging.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #29284
# Description
Creates 2 new models in the scope of flagging: how good are we at identifying "at risk" bookings vs. 1) the number of claims generated and 2) the number of submitted payouts?
This only applies for Protected Bookings in New Dash that have been completed (14 days after the check-out) with potential resolutions appearing in Resolutions Center.
The first table `int_flagging_booking_categorisation` contains all the heavy logic to categorise the bookings.
The second view `int_flagging_performance_analysis` computes standard binary classification scores, for the 2 possible ways of tracking.
Tables are already in prod to help you understand while reviewing. You'll see that the figures are still quite low, specially due to small amount of claims/submitted payouts. This makes the true positives being just... 1.
There's heavy test and documentation coverage to ensure there's no mistakes on the computation.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models. **Materialising as table the first model despite being just 1 record since otherwise tests takes ages**
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #29284
# Description
Adds more robust testing for Total Revenue and Revenue Retained Post-Resolutions in:
* int_mtd_vs_previous_year_metrics
* int_ytd_mtd_main_metrics_overview
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
More robust testing on key metrics
Related work items: #29227
# Description
Now that we finally have customers on CIH API
Nathan noticed an error on the report which we hadn't noticed before, the fees were not included in the reporting model and was previously being taken from the monthly model which did have them. So currently the monthly value is being displayed correctly but not the individual fees for each verification.
# Checklist
- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
cih api fix
Related work items: #29225
# Description
As raised by Kayla, Retained metrics are not computed correctly in AM Account Margin report, see screenshot:
Issue lies in the fact that Guest Payments contains more dimensions than the rest of the models. This is generating duplicates in the combination_of_sources CTE, but these are effectively hidden because of the sum() applied of the metrics.
In the case of a Deal having in the same day Guest Payments with different dimensions from those selected, AND on the same day having a value on Invoiced Revenue or Host Resolutions, then the Invoiced Revenue or Host Resolutions would be duplicated.
This PR aims to fix this issue, and also adds a few tests on the base model for AM Account Margin to ensure this is kind of error raises an alert.
I'll be debugging other affected models and increase the robustness test-wise; as well as gathering the full impact.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #29227
# Description
Small changes to exclude "irrelevant" data. All data is relevant I guess but this is less.
* Exclude churned accounts after 3 months.
* Exclude any data after 24 months.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #28998
# Description
Added `has_upgraded_screening_services_business_type`
So in the report we can filter by this because currently all bookings should have `has_screening_services_business_type` with BASIC SCREENING
# Checklist
- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
upgraded screening
Related work items: #28640
# Description
Creates a new model for reporting. It includes any relevant metric at Deal level in a similar format as we do for `mtd_aggregated_metrics`. Additionally, there's few Deal attributes - from Hubspot, segmentations, lifecycles, etc.
In order to dynamically choose which metrics are relevant on a Deal level, I modified the configuration in `int_mtd_aggregated_metrics` so the extraction is under control.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models. **Worth discussing the possibilities to include indexes in the future**
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #28998
# Description
To be able to compute By Deal KPIs similarly as we do for Main KPIs we need to propagate the "By Deal" dimension first in intermediate. This does not impact reporting since there's a macro that already cuts the dimensions to be displayed, but I modified the name so it's clearer.
Changes:
* In `int_mtd_vs_previous_year_metrics`, I added a new `dimension_list`. This is applied to all initial CTEs that filter by dimension. Note that I added here the `by_deal` dimension.
* Modified the name of `get_kpi_dimensions_for_production` to `get_main_kpis_dimensions_for_production`. Now it's more explicit that this is the configuration for Main KPIs reporting. This affects both `business_kpis_configuration` and it's usage in `mtd_aggregated_metrics`.
* Modified the tests in `int_mtd_vs_previous_year_metrics` and `int_mtd_aggregated_metrics` to include the new dimension `by_deal`.
* It seems by adding this now autoformatting works again on this model! I'll tag all cases that are just because of autoformatting.
MD5 trick has been applied in `reporting.mtd_aggregated_metrics`.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #28998
# Description
Removes old onboarding mrr models, that are no longer used.
dbt compiles correctly and no deprecated warning is showing anymore.
This finishes the refactor
Related work items: #28949
# Description
Puts in production the new KPIs model for Onboarding MRR.
The change has been validated with MD5 trick.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #28949
# Description
Creates a single model in KPIs, aggregated by dimension/dimension value, in a monthly basis; named: `int_kpis__agg_monthly_onboarding_mrr`.
Attention! This substitutes both current models, namely `int_monthly_onboarding_mrr_per_deal` and `int_mtd_agg_onboarding_mrr_revenue`. However, both models are currently used to retrieve data for Main KPIs. I just combined both into one, so it will simplify also the gathering of data later on.
Note that this model is special in the sense that the onboarding mrr per deal is computed for global, listing segmentation and billing country dimensions; while the total onboarding mrr is only done for global and listing segmentation (as it's based on the number of listings segmentation for the total compute).
This has been tested with dbt audit helper and md5. This has been a bit more complex since it's comparing 1 new model vs 2 existing models. For reference, this is the md5 comparison used:
```
SELECT md5(array_agg(md5((t1.*)::varchar))::varchar)
FROM (
SELECT
mrr.date,
mrr.dimension,
mrr.dimension_value,
mrr.expected_mrr_per_deal,
total.number_of_new_deals,
total.expected_mrr as total_expected_mrr
FROM intermediate.int_monthly_onboarding_mrr_per_deal mrr
left join intermediate.int_mtd_agg_onboarding_mrr_revenue total
on mrr.date = total.date
and mrr.dimension = total.dimension
and mrr.dimension_value = total.dimension_value
ORDER BY date, dimension, dimension_value
) AS t1
union all
SELECT md5(array_agg(md5((t2.*)::varchar))::varchar)
FROM (
SELECT *
FROM intermediate.int_kpis__agg_monthly_onboarding_mrr
ORDER BY date, dimension, dimension_value
) AS t2
```
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #27120, #28949
# Description
Tags as deprecated the following models:
* `int_monthly_onboarding_mrr_per_deal`
* `int_mtd_agg_onboarding_mrr_revenue`
Project compiles.
Related work items: #28949
# Description
Removes the models:
* int_monthly_12m_window_contribution_by_deal
* int_monthly_churn_metrics
as well as their entries in the schema files.
Project compiles and KPIs run works.
This closes stage 3 of the refactor.
# Checklist
- [ ] The edited models and dependants run properly with production data.
- [ ] The edited models are sufficiently documented.
- [ ] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Remove previous Churn models
Related work items: #28948
# Description
Small but important change.
Churn metrics will be propagated in production by sourcing intermediate/kpis rather than the cross models.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #28948
# Description
Creates a new model in intermediate/kpis called `int_kpis__agg_monthly_churn_contribution`. This follows the convention (aggregated = by dimension and dimension value) and monthly (1 dimension value per dimension per date where date is unique in month).
This single model aims to substitute, in the future, the 2 churn models simultaneously:
* `int_monthly_12m_window_contribution_by_deal`
* `int_monthly_churn_metrics`
Main changes:
* There's no longer a dependency with `int_monthly_aggregated_metrics_history_by_deal`. Rather, it gets the metrics and dimensions from wherever needed and nothing else.
* I also took the opportunity to clean any code that was not needed after combining both models, resulting in dropping a whole CTE.
* Updated schema description for clarification purposes.
The output of the new model is identical to the 2nd model, `int_monthly_churn_metrics`, confirmed with the md5 trick. Model runs in 1.45 seconds in my laptop so no performance issue (it's technically faster than running the 2 current models which is around 6.3 seconds but whatever).
Note that this has NO effect yet on production. The switch will be handled in a separated PR.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #28948
# Description
Extracts Integration Type (OSL, PMS, AutoHost, other API stuff) from Hubspot Deals.
This is needed for the categorisation of Deals in the scope of Old Dash to New Dash migration.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #28987
# Description
According to our previous conversation, modified the values for active listings and how they will be shown on the report's funnel.
_This changes will crash the report for now, I was thinking on adding a **work in progress** warning to both Overview and User Details tab in the meantime. After this the report should be soon ready_
Now we are going to have:
- **total_listings**
- **total_active_listings**
- **total_active_listings_with_active_product_bundle**
- **total_active_listings_with_active_product_bundle_with_paid_service**
Also fixed an aggregation mishap I had with the created bookings aggregated models
# Checklist
- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Update to listing count in new dash overview
Related work items: #28640
# Description
Churn models being tagged as deprecated for next week, for information.
# Checklist
- [ ] The edited models and dependants run properly with production data.
- [ ] The edited models are sufficiently documented.
- [ ] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Deprecation of Churn Models
Related work items: #28948
# Description
Decouples Churn/Account Margin model of `int_monthly_aggregated_metrics_history_by_deal_by_time_window` from the model `int_monthly_aggregated_metrics_history_by_deal`.
Changes:
* Removes dependency and reads from KPIs models.
* Removes unused deal stage variable
* Jinja template for time windows with dynamic selection to DRY
This has been tested with dbt audit tools and the md5 for pure refactoring, so output is exactly the same.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #28947
# Description
Adds audit helper.
Removes computation of Total and Retained Revenue from cross models.
I've tested audit helper and the trick with the hashes, as discussed in the demo.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them. **Except for today's outlier detector!**
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #28946
# Description
Adds 4 models for Total and Retained Revenue:
* 2 Metric models on Monthly and MTD
* 2 Aggregated models on Monthly and MTD
This follows the standard for KPIs. First manual checks vs data look ok, although I'll run the proper test analysis with the dbt audit tool in the next PR when merging into the current production tables.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #28946
