# Description
Changes (only in intermediate):
* Applies sqlfmt in KPIs source models (for some of them it was already applied). Specifically, the 3 Core models ONLY contains formatting changes
* Adds `main_deal_name` and `main_billing_country_iso_3_per_deal` in `int_monthly_aggregated_metrics_history_by_deal`
* Adds the 2 new fields in the schema entry of `int_monthly_aggregated_metrics_history_by_deal`, including the dbt test not null in the deal name.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #18911, #19083
# Description
Changes:
* Adding `main_deal_name` and `main_billing_country_iso_3_per_deal` in `int_dates_by_deal` model.
* Documents the 2 new fields. Also, ensures `main_deal_name` is not null
* Removes `id_deal not null` condition since it's enforced on the inner join with `int_core__deal`
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #18911, #19083
# Description
Created a mapping for some users that have a different id in Cosmos so we can join them using data that Ray shared with me, with this we can get all currency information for each user
# Checklist
- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [x] I have checked for DRY opportunities with other models and docs.
- [x] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #20125
# Description
This PR is a pure refactor (as in, doesn't change the output of the model at all).
The only purpose of the changes in this PR is to improve the performance on the model. The strategy to achieve this has been to inline the basic CTEs of the model (inline=replace references to the CTE with direct `ref` inside the model).
This works because it breaks an optimization fence.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #20835
# Description
Adds a new dbt test that will fail if the maximum date of a given column is different from yesterday. It uses `current_date`.
Additionally, I changed the test `kpis_global_metrics_outlier_detection` on the name of the parameter `detector_strength` to `detector_tolerance`, as a higher value of the parameter indicates that will be less likely to raise an alert.
Verified in local that tests passes if the execution is normal. Verified in local that the tests fails if manually deleting the latest date in the table.
# Checklist - Does not apply
- [ ] The edited models and dependants run properly with production data.
- [ ] The edited models are sufficiently documented.
- [ ] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #20824
# Description
I added verification status to be able to show the amount of bookings that have been approved vs not approved (rejected, flagged, or missing info)
# Checklist
- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [x] I have checked for DRY opportunities with other models and docs.
- [x] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #20125
# Description
guesty_verifications to reporting
# Checklist
- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [x] I have checked for DRY opportunities with other models and docs.
- [x] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #20125
# Description
Before deploying KPIs by Billing Country, we spotted some issues that were basically increases on the volumes of any metric on the by billing country dimension that was based on Deal. This means, `int_core__mtd_deal_metrics` and `int_xero__mtd_invoicing_metrics`.
This PR changes the following:
* Now the 2 abovementioned models depend on the `int_core__deal` model, instead of `int_core__user_host` (thus removing duplicated stuff)
* Now all models use the main billing country at deal level, instead of doing it so at host level. The reason is that some small amount of hosts that share the same deal can have a different billing country. To avoid weird stuff, everything points to this simplification - that in general, it's not a massive change in the output.
* In order to do so easily, the 3 main billing country per deal fields have been propagated to `int_core__user_host`
To exemplify the solution, find here a snapshot of the differences in behavior:
```
select
dimension,
sum(deals_booked_in_month) as deals_booked_1,
sum(deals_booked_in_6_months) as deals_booked_6,
sum(deals_booked_in_12_months) as deals_booked_12,
sum(total_revenue_in_gbp) as total_revenue,
sum(xero_operator_net_fees_in_gbp) as operator_revenue,
sum(xero_booking_net_fees_in_gbp) as booking_fees,
sum(xero_listing_net_fees_in_gbp) as listing_fees,
sum(xero_verification_net_fees_in_gbp) as verification_fees,
sum(total_guest_revenue_in_gbp) as guest_revenue,
sum(xero_waiver_paid_back_to_host_in_gbp) as waiver_paid_back_to_hosts,
sum(waiver_net_fees_in_gbp) as waiver_net_fees
from intermediate.int_mtd_vs_previous_year_metrics
where date in ('2024-01-31')
group by 1
order by 1
```
Production:
vs.
Local:
Keep in mind that still Global dimension can be greater than any other dimension aggregated since not all users have a deal. Mismatches between the other 2 dimensions might be linked to the dump.
Commits are meaningful and help navigate in the changes.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #20823
# Description
new model with Guesty verifications for business overview report, this one is a lot more simple since it only charges per nights of the booking. Though there is a slight problem that right now there are some verifications with bookings with 0 nights, those are being filtered out for the moment so they don't crash the model tests.
# Checklist
- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [x] I have checked for DRY opportunities with other models and docs.
- [x] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #20125
# Description
Creates a new master table containing the Deals. At this stage, the information is quite limited - I only included those fields that are useful for fixing the KPIs issues. To be enriched later on.
# Checklist
- [ ] The edited models and dependants run properly with production data. **Question here. Do I need to have the models materialised in local to see the dependants on DBT Docs? I didn't see any uses of the previous model int_core__deal_id_master_list.**
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #20822
# Description
This PR:
- Adds half-decent docs to `stg_edeposit__verifications` and tests. I say half-decent because I would describe our tests as "as strict as the backend guidance allows". But we can't do miracles, so it stays this way for now.
- Shifts a few column operations that were being done in the `int` layer into the `stg` layer.
- Also removes a couple of fields from `int` that were marked as deprecated by Ray. Would rather not have them at all beyond `stg`.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #20123
# Description
Adds Billing Country dimension in KPIs, but does not expose them to reporting yet.
Silly thing, based on the macros I built, I cannot make incremental changes unless changing all models. This will need to be adapted, happy to hear your thoughts on how we do it.
Additionally, I have lack of performance of the model `mtd_guest_payments_metrics`. It takes around 5 min to execute, but technically the end-to-end runs in one shoot without breaking.
It's a complex PR because it changes many files, but you will see that:
* It mostly changes the join conditions for the dimensions or the schema tests,
* I tried to be very careful and add things step-by-step in the commits.
Goal is NOT to complete the PR yet until we see how we can improve performance. I can say though that data end-to-end looks ok to me, but would benefit from checking with production data for the new dimension
Update 30th Aug
* Added a new commit that includes `id_user_host` in `int_core__verification_payments`. Happy to discuss if it makes sense or not. But it changes the execution from ~600 sec to ~6 sec because it avoids a massive repeated join with `verification_requests`.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models. **To check because of performance issues**
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19082
# Description
Exposes Billable Bookings metric for KPIs, both in the "global+dimension" view and in the "deal" view.
Metrics have already been created for a while. Exposing them now after the changes carried out in the model `int_core__booking_charge_events`. Based on the current quality of the data, I opted for "Est. Billable Bookings" to account for the fact that this is an estimation. If you don't feel comfortable with it, let's remove the "Est. ".
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #18111
# Description
edeposit fees per verification
I modified the model so it contains only the fees charged per verification so then we can do the grouping and filters in power bi depending on how it needs to be displayed.
# Checklist
- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [x] I have checked for DRY opportunities with other models and docs.
- [x] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #20125
# Description
Forces lower case to all id_users in staging. Removes hardcoded lower case in intermediate. Adapts readme to contemplate the lowering of id users.
I propose to merge, run in prod and run tests in prod as a proper evaluation method.
BTW, I only find one id_user_host that was in capital letters, so that's why probably we didn't care that much about this. Still, I prefer have things clean from the start!
```
select *
from staging.stg_core__booking scb
left join intermediate.int_core__unified_user icuu
on lower(scb.id_user_host) = lower(icuu.id_user)
where scb.id_user_host <> icuu.id_user
```
# Checklist
- [ ] The edited models and dependants run properly with production data. **All models run in stg, did not check all the dependants**
- [ ] The edited models are sufficiently documented. **Have not checked**
- [ ] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #20776
# Description
Removing coalesce from gbp conversion in `int_core__host_booking_fees`
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs. **Message sent in data team channel**
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
