# Description
New model to replace in the host_fees report which is now including currencies
# Checklist
- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.
# Other
- [ ] Check if a full-refresh is required after this PR is merged.
Related work items: #18875
This PR creates a new model int_core__mtd_accommodation_segmentation that provides the customer segments based on listing activity:
- 0
- 1-5
- 6-20
- 21-60
- 61+
For end of April 2024, the volume distribution on number of deals and total listings booked is:
For information, I estimate that around 3% of listings with bookings are missed, according to the data displayed in the KPIs for 30th April 2024. This is because we enforce deal-based categorisation (same happens with the deal view, anyway)
Related work items: #19325
After discussion with Pablo on the fact that Deposits are only with status "Paid" for a given time before they get Cancelled or Refunded, we just believe it's best to remove the Deposits amount from the Guest Payments metric. In any case, this does not represent a Revenue source... This was discovered while doing the data quality assessment for revenue figures ([here](https://www.notion.so/knowyourguest-superhog/Data-quality-assessment-DWH-vs-Finance-revenue-figures-6e3d6b75cdd4463687de899da8aab6fb))
Before, `total_guest_payments_in_gbp` was a standalone metric that computed any payment by the guest with status paid. We were computing revenue based on the `total_guest_income_in_gbp`, which mainly was the sum of waiver payments, deposit fees (not deposit itself!) and check in hero fees.
Mainly what I did is:
- remove the existing `total_guest_payments_in_gbp` in the source models (int_core__xxx_guest_payments_xxx)
- rename the already existing `total_guest_income_in_gbp` to `total_guest_payments_in_gbp`
Related work items: #18787, #18914
Exposes the following metrics on the "by deal" view of business kpis:
1. Invoiced Booking Fees
1. Invoiced Listing Fees
1. Invoiced Verification Fees
1. Invoiced Guesty Fees
1. Invoiced E-Deposit Fees
1. Deposit Fees
1. Waiver Amount Paid by Guests
1. Waiver Amount Paid back to Hosts
1. Waiver Net Fees
1. Check-In Hero Amount Paid by Guests
This PR does not impact the visualisation in the dashboard, meaning, metrics won't automatically appear for users and needs an additional change in the PBI reporting.
Related work items: #18914
Exposes the following metrics:
1. Invoiced Booking Fees
1. Invoiced Listing Fees
1. Invoiced Verification Fees
1. Invoiced Guesty Fees
1. Invoiced E-Deposit Fees
1. Deposit Fees
1. Waiver Amount Paid by Guests
1. Waiver Amount Paid back to Hosts
1. Check-In Hero Amount Paid by Guests
This PR is only for Global metrics, the one by deal id will follow separately.
Keep in mind that merging this PR will make the data appear in the report.
Related work items: #18914
Small refactor to follow up on last week's PR. We removed from the Guest Revenue models the host-takes-waiver aspect, thus these models are now only depending from Core. We just need to migrate it from cross to core.
One small detail as well, since we do not take into account at these models level the host-takes-waiver, technically, I would not call these models revenue but rather Guest Payments. This is why I also took the opportunity to apply this name.
Changes:
- `int_monthly_guest_revenue_by_deal` is now `int_core__monthly_guest_payments_history_by_deal`, and the location has changed from `intermediate.cross` to `intermediate.core`
- `int_mtd_guest_revenue_metrics` is now `int_core__mtd_guest_payments_metrics`, and the location has changed from `intermediate.cross` to `intermediate.core`
- Schema changes, moving these 2 models' documentation with the new naming from Cross to Core
- Provide continuity in following dependants: `int_mtd_vs_previous_year_metrics` and `int_monthly_aggregated_metrics_history_by_deal` now read from the 2 new models respectively. Additionally, the model alias has changed from `guest_revenue` to `guest_payments` to keep consistency.
This PR does not expose new metrics, but should keep the existing ones unaffected.
Related work items: #18914
Adding submetrics of guest revenue by deal:
- deposit_fees_in_gbp
- checkin_cover_fees_in_gbp
- waiver_payments_in_gbp
all of this adds up to
- total_guest_income_in_gbp
and the total_guest_revenue_in_gbp is now computed by subtracting waivers paid to hosts, coming from the invoiced model. This also affects the total revenue computation and the weighted metrics.
This PR it's completely equivalent to the [Adding submetrics of guest revenue](https://guardhog.visualstudio.com/Data/_git/data-dwh-dbt-project/pullrequest/2381?path=/models/intermediate/cross/int_mtd_guest_revenue_metrics.sql&_a=files), that was already merged for the global view. The only difference is that this one is for the deal-based part. It does not expose the metrics to the report.
A follow-up PR is expected to apply the correct naming conventions for guest revenue models.
I recommend checking the first commit, the one that applies the changes, and should be easily understandable with the link to the previous PR. There's a couple of additional commits that only affect formatting.
Related work items: #18787, #18914
Created new model with listings with check in hero active for the report so users can have access to this more granulated data
Related work items: #18190
Adding submetrics of guest revenue:
- deposit fees
- checkin cover fees
- waiver payments
all of this adds up to
- guest income
and the revenue is computed by subtracting waivers paid to hosts
Related work items: #18787
This PR exposes the following metrics to the Main KPIs business overview report, for both Global + By Deal view:
- Total Revenue
- Total Revenue per Booking Created
- Total Revenue per Guest Journey Created
- Total Revenue per Deals Booked in Month (does not apply on the by deal view)
- Total Revenue per Listings Booked in Month
- Invoiced Operator Revenue
- Host Resolution Payment Count
- Host Resolution Amount Paid
Keep in mind Global view will be displaying these metrics once this is merged. I also changed a bit the order of the metric display.
Note that Billable Bookings are not included.
I recommend to review by 1) checking the first commit. This is almost the same as the previous abandoned PR that @<Joaquin Ossa> you already checked on Tuesday. I added a second commit, to be checked later, which basically fixes some stupid issues that if one of the source of revenue is null, then total revenue is null. This is specially critical for the view by deal, since most of them do not have revenue from APIs - thus all total revenue figures were null...
Related work items: #18108, #18109, #18110, #18719
Computing and propagating APIs revenue metrics.
I retrieved the revenues linked to Guesty and e-deposits. The sum of those are considered the total API revenue at this stage.
These 3 metrics are available in upper layers (not exposed yet to the report), just to fix the total revenue computation, which now includes APIs revenue
Related work items: #18719
Discussing with Jamie on how I can retrieve the information from Xero for E-deposit and Guesty, he told me that these go to a dedicated account. Since these are invoiced, they do not go to the bank transactions.
Specifically:
- 220 - E-Deposit Fees
- 219 - Guesty Fees
- 221 - Guesty Administration fee
After debugging the data (and getting confirmation from Jamie that indeed we were having invoices on e-deposit), I see that this info is available in the original json from staging. This PR aims just to retrieve these 3 fields:
- id_account
- account_code
- account_name
This will be used later on to retrieve the rest of API KPIs
Related work items: #18719
Propagates billable bookings kpis in intermediate. Does not expose any metric to the report.
Changes:
- Retrieval and computation of previous_year and relative_increment for global view (mtd models)
- Retrieval as is for deal view
Related work items: #18111
This PR computes KPIs for Billable Bookings for both views (Global, by deal id)
It's the 1st version, mainly because numbers are not fully in-line with Jamie's export and I'd like to understand why. It uses booking_charge_events, as suggested by Pablo. In the meantime, I'm debugging the differences based on the invoicing export tool that provides Finance first rough numbers before amendment.
In any case, it can be used for a first rough idea.
Related work items: #18111
This PR aims to propagate the invoicing metrics through the DWH. It does not expose them to users, yet.
This PR effectively computes the following metrics, for both the "global" view (MTD) and the "by deal" view (by_deal):
- Invoiced Operator Revenue
- Host Resolution Count of Payments
- Host Resolution Amount Paid
With these 3 new metrics, we're able to combine them with the existing ones to compute:
- Total Revenue
- Total Revenue per Booking Created
- Total Revenue per Guest Journey Created
- Total Revenue per Deal Booked in Month
- Total Revenue per Listings Booked in Month
You'll also note that I've included standalone metrics for booking fees, listing fees, verification fees and waiver payments. This will not be exposed in this batch 2, but based on the conversation with Finance, will clearly make it for batch 3. I just find it easier to add it now, since it's straight forward.
Main changes:
- `int_mtd_vs_previous_year_metrics` now computes all the above mentioned metrics
- `int_monthly_aggregated_metrics_history_by_deal` now computes all the above mentioned metrics, except Total Revenue per Deal Booked in Month since it does not make sense for the deal view. Additionally, I took the opportunity to include the missing metrics from listings (accommodations). The goal is not necessarily to display them, but at least compute it on our side.
Additional changes:
- In `int_xero__mtd_invoicing_metrics` and `int_xero__monthly_invoicing_history_by_deal`, there's a very silly name change to keep the same convention for fees: from `xero_operator_net_fees` to `xero_operator_net_fees_in_gbp`
- I applied additional changes in `int_monthly_aggregated_metrics_history_by_deal` with the goal to keep the same format as we have in `int_mtd_vs_previous_year_metrics`, this meaning:
1 - explicit alias naming (from `gj` to `guest_journeys`)
2 - keep a similar arrangement of metrics, and clearly separate scopes depending on the metric type
3 - Re-apply autoformatting
Related work items: #18108, #18109, #18110
Refactor mtd joins to improve performance, as stated in the ticket:
We noticed that some of the new models for MTD purposes (KPIs reporting) take quite a bit of time to run some simple joins.
The main reason is that there's a double join that can be simplified. The current state is:
```
from int_dates_mtd d
inner join
sometable t
on extract(year from t.table_date) = d.year
and extract(month from t.table_date) = d.month
and extract(day from t.table_date) <= d.day
```
and it can be changed to:
```
from int_dates_mtd d
inner join
sometable t
on date_trunc('month', t.table_date)::date = d.first_day_month
and extract(day from t.table_date) <= d.day
```
which is way faster, and keeps the same computation
Related work items: #18330
This PR creates 2 new models on intermediate, xero:
- int_xero__mtd_invoicing_metrics (global view)
- int_xero__monthly_invoicing_history_by_deal (by deal view)
This allows for the computation of host metrics (operators), as well as host resolution payments. This will enable in the future to compute total revenue and weighted revenue metrics.
The data displayed from the previous months in the mtd_invoicing_metrics is consistent with 1) revenue figures displayed in business overview for Host tab, as well as Guest tab for Waiver Payments; and 2) host payment figures displayed in the accounting reports.
**Note 1**: the variables at this stage are reused in these models, as well as many other Xero models. We still need to handle the refactor ticket on Xero related reports. This is not under the scope of this PR.
**Note 2**: we noticed that the strategy for mtd models to do a double year, month extraction join is badly performant. Actually, resolution payments subquery was not performing at all with this logic. So it has been changed to a date_trunc('month', related_date)::date = d.first_day_month strategy which works much faster. I'll add a standalone PR to refactor the remaining mtd models separately later
Related work items: #18108, #18110
This PR aims to expose the new metrics to the business KPIs report.
The new metrics exposed are, for the global and the by deal view:
- Guest Revenue
- Guest Revenue per Guest Journey Completed
- Guest Revenue per Guest Journey with Payment
- Guest Payments
- Guest Payments per Guest Journey Completed
- Guest Payments per Guest Journey with Payment
- Guest Journey with Payment
- Guest Journey Payment Rate
Changes:
- Silly change on the naming in the by deal view of `payment_rate_guest_journey` to be consistent with the global view.
- Silly change that I miss some GJ payment metric for the view by deal id.
- Added a new number format called `currency_gbp` - only for monetary metrics, available in the schema files
- Usual procedure to publish metrics: for global metrics, add them in the `int_mtd_aggregated_metrics`. I also changed the order of display.
- **Important**: to avoid displaying revenue figures until Xero invoicing is handled, I created a macro called `is_date_before_previous_month` that is called in the reporting equivalent models: `mtd_aggregated_metrics` in the where section and in the `monthly_aggregated_metrics_history_by_deal` as a case-when.
This should allow to expose all new metrics, and enable the publishing of a new update of the business kpis!
Related work items: #18107
This PR aims to propagate the computation of guest revenue model into the aggregated models.
Changes:
- I changed the logic on `int_mtd_guest_revenue_metrics` since it was still using the old computation of the relative increment within the same model. Basically, I removed it (last part of the query). The rest of changes in this model are just formatting.
- I also applied the formatting in the int_mtd_vs_previous_year_metrics, mainly changing the macro calls from **'**xyz**'** to **"**xyz**"**.
What's new:
- `int_mtd_guest_revenue_metrics` into `int_mtd_vs_previous_year_metrics`, by adding `total_guest_revenue_in_gbp` and `total_guest_payments_in_gbp`. Additionally, with the new logic, for the first time we're able to compute **weighted metrics** coming from different sources. Specifically, it computes the weighted measures per guest journey completed and guest journey with payment.
- Similar behavior on the 'by deal', adding `int_monthly_guest_revenue_history_by_deal` into `int_monthly_aggregated_metrics` and similar computation
This model does not affect the exposing logic still, meaning these metrics won't be exposed in the report. This will come in a separated PR.
Related work items: #18107
Changing naming to follow convention.
This PR has the following changes:
- the model `int_core__mtd_aggregated_metrics` has been moved to cross and changed the name to `int_mtd_aggregated_metrics`
- the model `int_core__monthly_aggregated_metrics_history_by_deal` has been moved to cross and changed the name to `int_monthly_aggregated_metrics_history_by_deal`
- the reporting models `core__mtd_aggregated_metrics` and `core__monthly_aggregated_metrics_history_by_deal` now source the `int_mtd_aggregated_metrics` and `int_monthly_aggregated_metrics_history_by_deal` to avoid breaking the production dashboard
- the reporting models have been duplicated from core into general with the correct names, i.e., `mtd_aggregated_metrics` and `monthly_aggregated_metrics_history_by_deal`
- Documentation has been moved in intermediate and replicated in reporting, adding comments on the currently in use models that are going to die soon.
This will allow for a transition of the PBI dashboard from one source to another. Exposures file still not touched since technically the report is still sourcing the 'legacy' models. Documentation of the refactor here: https://www.notion.so/knowyourguest-superhog/Refactoring-Business-KPIs-5deb6aadddb34884ae90339402ac16e3
Related work items: #18202
