# Description
Created address_validations in intermediate for check in hero
# Checklist
- [X] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [x] I have checked for DRY opportunities with other models and docs.
- [x] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #17069
# Description
Modifies `int_core__mtd_guest_payments_metrics` to include the customer segmentation based on listings. `schema.yaml` is also affected including new fields, tests and apply the proper naming (from guest revenue to guest payments). I also modified a silly naming that was referring to deals to refer to listings/accommodations, my bad.
Added the macro to retrieve the production dimension in `int_core__mtd_vs_previous_year_metrics` to avoid propagating this upwards and messing up with the data display.
Overall, follows a similar strategy as we did for Booking, Guest Journey, Deal and Accommodation metrics. For reference, [here's the previous PR on Accommodations](https://guardhog.visualstudio.com/Data/_git/data-dwh-dbt-project/pullrequest/2575?_a=overview).
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19325
# Description
Takes into account @<Pablo Martín> 's feedback from the previous PR, slightly modified. This PR separates 1) the dimensions while developing vs. 2) the dimensions once these are available for production. This are within the same file of macro configuration for KPIs, namely `business_kpis_configuration`.
End-goal, all CTEs in `int_mtd_vs_previous_year_metrics` will read from this new macro `get_kpi_dimensions_for_production`, so eventually we won't need any hardcode once we want to add new dimensions. In the meantime, I'll be adding this new line for each PR (still 2 missing :D)
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [ ] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19325
# Description
Modifies `int_core__mtd_accommodation_metrics` to include the customer segmentation based on listings. `schema.yaml` is also affected including new fields and tests. Hardcoded `int_core__mtd_vs_previous_year_metrics` to avoid propagating this upwards and messing up with the data display.
Overall, follows a similar strategy as we did for Booking, Guest Journey and Deal metrics. For reference, here's [the previous PR on Deal](https://guardhog.visualstudio.com/Data/_git/data-dwh-dbt-project/pullrequest/2534). I noticed that I mixed the schema tests of Deals and Accommodations, this PR should fix both.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19325
# Description
Modifies `int_core__mtd_deal_metrics` to include the customer segmentation based on listings. `schema.yaml` is also affected including new fields and tests. Hardcoded `int_core__mtd_vs_previous_year_metrics` to avoid propagating this upwards and messing up with the data display.
Overall, follows a similar strategy as we did for Booking and Guest Journey metrics. For reference, [here's the previous PR on GJ](https://guardhog.visualstudio.com/Data/_git/data-dwh-dbt-project/pullrequest/2533).
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19325
# Description
Modifies `int_core__mtd_guest_journey_metrics` to include the customer segmentation based on listings. `schema.yaml` is also affected including new fields and tests. Hardcoded `int_core__mtd_vs_previous_year_metrics` to avoid propagating this upwards and messing up with the data display.
Overall, follows a similar strategy as we did for Booking metrics.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19325
# Description
_Describe your motivation and changes here._
# Checklist
- [ ] The edited models and dependants run properly with production data.
- [ ] The edited models are sufficiently documented.
- [ ] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
# Description
Added relative_increment_with_sign_format for special formatting in PBI
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Added relative_increment_with_sign_format
Reverts !2524
Related work items: #19559
# Description
What's new:
- Creation of `get_kpi_dimensions`: new macro to have a single point of source of configuration for dimensions for the KPIs. It's a way to enforce global variables on-demand. I kind of like this approach and we could do it for Xero models as well :)
- Modification of `int_core__mtd_booking_metrics` and `int_dates_mtd_by_dimension`: removal of duplicated code within the dimension context. Uses Jinja code and applies different configurations depending on the dimension chosen. Still, different metrics are placed in different CTEs. I believe it might be possible to also configure metrics BUT at the cost of over-complexifying the macro logic, so I wouldn't go for it at this stage.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] **I have checked for DRY opportunities with other models and docs.**
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19325
# Description
This is a first idea of how I'd like to add dimensionality in the KPIs for the mtd models. For the moment, I keep deal_id apart, so I just touch the "mtd" models, that so far only contained "global" metrics.
In this case I include the listing segmentation (0, 1-5, 6-20, etc) in the bookings. To do this, I created 2 new fields: dimension and dimension_values.
I also created a "master" table with `date` - `dimension` - `dimension_value` called `int_dates_mtd_by_dimension`
Important notes:
- I force a hardcode in `int_mtd_vs_previous_year_metrics`. This is to not break production.
- You will notice how repetitive the code is starting to look. My intention with this PR is that we are happy with this approach on the naming, the strategy for joins, etc. If that's ok, next step is going to be doing macros on top. Think of the state of `int_core__mtd_booking_metrics` as the "compiled version" of the macro that should come afterwards.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19325
# Description
Adds 2 new tables:
- `int_core__user_role`: contains the relationship of a given user has a role.
- `int_core__user_host`: based on the previous table, it selects the users and main information from those users that are considered as hosts according to the role they have.
Note: I needed to change the test in stg. A user, generally, can have no role, one role, or multiple roles. Thus we cannot propagate this information in the unified_user model.
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #19513
# Description
Materialise int_dates_mtd and int_dates_by_deal as tables. This should improve the run speed as seen in local by quite a bit, and hopefully provide a better starting point for adding new dimensionality on business kpis.
I also documented these 2 models, that were missing :)
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [] Check if a full-refresh is required after this PR is merged.
Related work items: #19514
# Description
I modified the model of duplicated_bookings, I saw it had some errors and data that was not being used
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #18875
This PR creates a new model int_core__mtd_accommodation_segmentation that provides the customer segments based on listing activity:
- 0
- 1-5
- 6-20
- 21-60
- 61+
For end of April 2024, the volume distribution on number of deals and total listings booked is:
For information, I estimate that around 3% of listings with bookings are missed, according to the data displayed in the KPIs for 30th April 2024. This is because we enforce deal-based categorisation (same happens with the deal view, anyway)
Related work items: #19325
After discussion with Pablo on the fact that Deposits are only with status "Paid" for a given time before they get Cancelled or Refunded, we just believe it's best to remove the Deposits amount from the Guest Payments metric. In any case, this does not represent a Revenue source... This was discovered while doing the data quality assessment for revenue figures ([here](https://www.notion.so/knowyourguest-superhog/Data-quality-assessment-DWH-vs-Finance-revenue-figures-6e3d6b75cdd4463687de899da8aab6fb))
Before, `total_guest_payments_in_gbp` was a standalone metric that computed any payment by the guest with status paid. We were computing revenue based on the `total_guest_income_in_gbp`, which mainly was the sum of waiver payments, deposit fees (not deposit itself!) and check in hero fees.
Mainly what I did is:
- remove the existing `total_guest_payments_in_gbp` in the source models (int_core__xxx_guest_payments_xxx)
- rename the already existing `total_guest_income_in_gbp` to `total_guest_payments_in_gbp`
Related work items: #18787, #18914
Exposes the following metrics:
1. Invoiced Booking Fees
1. Invoiced Listing Fees
1. Invoiced Verification Fees
1. Invoiced Guesty Fees
1. Invoiced E-Deposit Fees
1. Deposit Fees
1. Waiver Amount Paid by Guests
1. Waiver Amount Paid back to Hosts
1. Check-In Hero Amount Paid by Guests
This PR is only for Global metrics, the one by deal id will follow separately.
Keep in mind that merging this PR will make the data appear in the report.
Related work items: #18914
Small refactor to follow up on last week's PR. We removed from the Guest Revenue models the host-takes-waiver aspect, thus these models are now only depending from Core. We just need to migrate it from cross to core.
One small detail as well, since we do not take into account at these models level the host-takes-waiver, technically, I would not call these models revenue but rather Guest Payments. This is why I also took the opportunity to apply this name.
Changes:
- `int_monthly_guest_revenue_by_deal` is now `int_core__monthly_guest_payments_history_by_deal`, and the location has changed from `intermediate.cross` to `intermediate.core`
- `int_mtd_guest_revenue_metrics` is now `int_core__mtd_guest_payments_metrics`, and the location has changed from `intermediate.cross` to `intermediate.core`
- Schema changes, moving these 2 models' documentation with the new naming from Cross to Core
- Provide continuity in following dependants: `int_mtd_vs_previous_year_metrics` and `int_monthly_aggregated_metrics_history_by_deal` now read from the 2 new models respectively. Additionally, the model alias has changed from `guest_revenue` to `guest_payments` to keep consistency.
This PR does not expose new metrics, but should keep the existing ones unaffected.
Related work items: #18914
Adding submetrics of guest revenue by deal:
- deposit_fees_in_gbp
- checkin_cover_fees_in_gbp
- waiver_payments_in_gbp
all of this adds up to
- total_guest_income_in_gbp
and the total_guest_revenue_in_gbp is now computed by subtracting waivers paid to hosts, coming from the invoiced model. This also affects the total revenue computation and the weighted metrics.
This PR it's completely equivalent to the [Adding submetrics of guest revenue](https://guardhog.visualstudio.com/Data/_git/data-dwh-dbt-project/pullrequest/2381?path=/models/intermediate/cross/int_mtd_guest_revenue_metrics.sql&_a=files), that was already merged for the global view. The only difference is that this one is for the deal-based part. It does not expose the metrics to the report.
A follow-up PR is expected to apply the correct naming conventions for guest revenue models.
I recommend checking the first commit, the one that applies the changes, and should be easily understandable with the link to the previous PR. There's a couple of additional commits that only affect formatting.
Related work items: #18787, #18914
Adding submetrics of guest revenue:
- deposit fees
- checkin cover fees
- waiver payments
all of this adds up to
- guest income
and the revenue is computed by subtracting waivers paid to hosts
Related work items: #18787
This PR exposes the following metrics to the Main KPIs business overview report, for both Global + By Deal view:
- Total Revenue
- Total Revenue per Booking Created
- Total Revenue per Guest Journey Created
- Total Revenue per Deals Booked in Month (does not apply on the by deal view)
- Total Revenue per Listings Booked in Month
- Invoiced Operator Revenue
- Host Resolution Payment Count
- Host Resolution Amount Paid
Keep in mind Global view will be displaying these metrics once this is merged. I also changed a bit the order of the metric display.
Note that Billable Bookings are not included.
I recommend to review by 1) checking the first commit. This is almost the same as the previous abandoned PR that @<Joaquin Ossa> you already checked on Tuesday. I added a second commit, to be checked later, which basically fixes some stupid issues that if one of the source of revenue is null, then total revenue is null. This is specially critical for the view by deal, since most of them do not have revenue from APIs - thus all total revenue figures were null...
Related work items: #18108, #18109, #18110, #18719
Computing and propagating APIs revenue metrics.
I retrieved the revenues linked to Guesty and e-deposits. The sum of those are considered the total API revenue at this stage.
These 3 metrics are available in upper layers (not exposed yet to the report), just to fix the total revenue computation, which now includes APIs revenue
Related work items: #18719
Discussing with Jamie on how I can retrieve the information from Xero for E-deposit and Guesty, he told me that these go to a dedicated account. Since these are invoiced, they do not go to the bank transactions.
Specifically:
- 220 - E-Deposit Fees
- 219 - Guesty Fees
- 221 - Guesty Administration fee
After debugging the data (and getting confirmation from Jamie that indeed we were having invoices on e-deposit), I see that this info is available in the original json from staging. This PR aims just to retrieve these 3 fields:
- id_account
- account_code
- account_name
This will be used later on to retrieve the rest of API KPIs
Related work items: #18719
Propagates billable bookings kpis in intermediate. Does not expose any metric to the report.
Changes:
- Retrieval and computation of previous_year and relative_increment for global view (mtd models)
- Retrieval as is for deal view
Related work items: #18111
This PR computes KPIs for Billable Bookings for both views (Global, by deal id)
It's the 1st version, mainly because numbers are not fully in-line with Jamie's export and I'd like to understand why. It uses booking_charge_events, as suggested by Pablo. In the meantime, I'm debugging the differences based on the invoicing export tool that provides Finance first rough numbers before amendment.
In any case, it can be used for a first rough idea.
Related work items: #18111
This PR aims to propagate the invoicing metrics through the DWH. It does not expose them to users, yet.
This PR effectively computes the following metrics, for both the "global" view (MTD) and the "by deal" view (by_deal):
- Invoiced Operator Revenue
- Host Resolution Count of Payments
- Host Resolution Amount Paid
With these 3 new metrics, we're able to combine them with the existing ones to compute:
- Total Revenue
- Total Revenue per Booking Created
- Total Revenue per Guest Journey Created
- Total Revenue per Deal Booked in Month
- Total Revenue per Listings Booked in Month
You'll also note that I've included standalone metrics for booking fees, listing fees, verification fees and waiver payments. This will not be exposed in this batch 2, but based on the conversation with Finance, will clearly make it for batch 3. I just find it easier to add it now, since it's straight forward.
Main changes:
- `int_mtd_vs_previous_year_metrics` now computes all the above mentioned metrics
- `int_monthly_aggregated_metrics_history_by_deal` now computes all the above mentioned metrics, except Total Revenue per Deal Booked in Month since it does not make sense for the deal view. Additionally, I took the opportunity to include the missing metrics from listings (accommodations). The goal is not necessarily to display them, but at least compute it on our side.
Additional changes:
- In `int_xero__mtd_invoicing_metrics` and `int_xero__monthly_invoicing_history_by_deal`, there's a very silly name change to keep the same convention for fees: from `xero_operator_net_fees` to `xero_operator_net_fees_in_gbp`
- I applied additional changes in `int_monthly_aggregated_metrics_history_by_deal` with the goal to keep the same format as we have in `int_mtd_vs_previous_year_metrics`, this meaning:
1 - explicit alias naming (from `gj` to `guest_journeys`)
2 - keep a similar arrangement of metrics, and clearly separate scopes depending on the metric type
3 - Re-apply autoformatting
Related work items: #18108, #18109, #18110
Refactor mtd joins to improve performance, as stated in the ticket:
We noticed that some of the new models for MTD purposes (KPIs reporting) take quite a bit of time to run some simple joins.
The main reason is that there's a double join that can be simplified. The current state is:
```
from int_dates_mtd d
inner join
sometable t
on extract(year from t.table_date) = d.year
and extract(month from t.table_date) = d.month
and extract(day from t.table_date) <= d.day
```
and it can be changed to:
```
from int_dates_mtd d
inner join
sometable t
on date_trunc('month', t.table_date)::date = d.first_day_month
and extract(day from t.table_date) <= d.day
```
which is way faster, and keeps the same computation
Related work items: #18330
