20250605-01 - Overrepresentation of Host Resolutions Payments

Managed by Uri

Summary

A data bug led to Host Resolutions Payments being overrepresented across various reports. This inflated values for related KPIs and metrics, particularly affecting:

Business Overview: Overstated Host Resolution Payouts, understated revenue retained post-resolutions.
Accounting Reports: Overstated Host Resolution Payments.
Account Management Reports: Skewed margins and growth data.

The bug caused a net overstatement of ~£7.2K, with the majority of the impact (~£6.5K) occurring in the first five days of June.

Timelines

Started at: 28th April 2025
Detected at: 5th June 2025, 11:02 UTC
Acknowledged at: 5th June 2025, 11:06 UTC (4 minutes post-detection)
Mitigated at: 5th June 2025, 11:38 UTC (36 minutes post-detection)
Resolved at: 5th June 2025, 11:39 UTC (37 minutes post-detection)

Impact

Host Resolutions Payments were showing higher values than the reality. This affected directly Host Resolution Payments details, aggregations and dependant metrics, which included Revenue Retained Post-Resolutions.

Temporal affectation

The over representation of Host Resolutions Payments was effectively reported in Power BI between 28th April 2025 and 5th June 2025.

The majority of the impact was during June, accounting for 6.5K GBP. The issue went unnoticed until 5th June, mostly because the relative impact of the mismatch vs. the days in month was much more evident than in previous months.

Detail is available in this table, extracted on June 5th:

Month	Over represented amount (GBP)	GBP/day
April 2025	-114	-3.8
May 2025	-562	-18.1
June 2025	-6,520	-1,304 (only first 5 days)

Reporting affectations

In Business Overview:

Main KPIs:
- Business Targets - Host Resolution Payouts was showing higher values.
- The following metrics available in the report were showing higher values, in absolute terms:
  - Host Resolutions Payouts
  - Host Resolutions Payouts per Booking Created
  - Host Resolutions Payment Count
  - Host resolutions Payment Count per Created Booking Rate
- The following metrics available in the report were showing lower values, in absolute terms:
  - Revenue Retained Post-Resolutions
  - Revenue Retained Post-Resolutions Rate

In Accounting Reports:

Resolutions Host Payments: all tabs were affected, Host Resolutions Payments were showing higher values in absolute terms.

In Account Management Reporting:

Account Margin: Similar impact as for Main KPIs.
Account Performance: Similar impact as for Main KPIs.
Churn Report: Revenue Retained Post-Resolutions was showing lower values.
Account Growth: Revenue Retained Post-Resolutions was showing lower values.

Timeline

Time (UTC)	Event
2025-04-25	A new model called `int_xero__host_resolutions_payments` is released in DWH, which reads from Xero. This aims to unify Host Resolutions Payments from two sources: Bank Transactions and Credit Notes. Prior to this, only Bank Transactions were considered for Host Resolutions Payments. Linked Pull Request !5015.
2025-04-28	New logic is applied for KPIs purposes, effectively affecting Main KPIs and Account Management in Power BI. Linked Pull Request !5071.
2025-04-29	New logic is applied for Xero reporting, effectively affecting Resolutions Host Payments in Power BI. Linked Pull Request !5083.
2025-06-04 06:41	Chloe messages Uri saying that there seems to be a bug in the dial of Host Resolution Payouts in Business Targets (Main KPIs). Uri explains how the projection is based on run-rate, and that there’s no bug in this computation.
However, the 15K GBP in Actual payouts are indeed bugged, as figures are higher than what should have been in reality. This however goes unnoticed.
2025-06-05 11:02	Chloe Lorusso rightly points out that Resolution Payouts are incorrect, as some lines seem duplicated in Power BI → Accounting Reports → Resolutions - Host Payments → Details
2025-06-05 11:06	Uri starts the investigation at Data side.
2025-06-05 11:14	Issue is identified: Resolution payments as Credit Notes are not being filtered by status. This means that status such as DELETED are considered, which result in over consideration of Host Resolutions amount paid.
2025-06-05 11:36	A fix is applied in `int_xero__host_resolutions_payments` to only include status AUTHORISED and PAID for Credit Notes. No change is applied in Bank Transactions, as it already considered only AUTHORISED.
2025-06-05 11:38	Incident is mitigated: any downstream dependant of `int_xero__host_resolutions_payments` is run in DWH production, manually, from local command.
2025-06-05 11:39	Incident is resolved: Bugfix has been pushed to the master branch of DWH in commit 08678427.

Root Cause(s)

The root cause was a bug within a DWH model from Xero that combines Host Resolutions Payments appearing as Bank Transactions and Credit Notes, namely int_xero__host_resolutions_payments. This model was put in production late April 2025.

Prior to the existence of this unified model, only Bank Transactions were considered. Upon adding Credit Notes, these were only being filtered by document type corresponding to ACCRECCREDIT, while no filter existed on document status. This means that Credit Notes with DELETED and VOIDED status were also considered.

Resolution and recovery

Resolution starts on June 5th, at around 11:06 UTC, when Uri deep-dives into the Host Resolution Payments line items that Chloe raised as duplicated a few minutes before. Data team gets notified of an ongoing incident.

Taking a few examples from DWH data, it’s clear that the issue lies in not filtering out Credit Notes with DELETED status.

At the moment of applying the fix, it’s noticed that a status named VOIDED also exists. Due to low amount of volume and the fact we never actually considered this status before for other Xero models, Uri unilaterally decides to discard it as well, despite it’s worth to check with Finance if these should be included. The detail of Host Resolutions Payments as Credit Notes can be seen below:

Document Status	Amount Paid (GBP)
PAID	-116,815
AUTHORISED	-8,613
DELETED	-6,803
VOIDED	-391

Thus, the fix effectively only includes Credit Notes with status PAID and AUTHORISED.

Fix is tested locally and then run in DWH production. Once finished, Uri refreshes the Power BI - Host Resolutions Payments and sees how the buggy lines have disappeared. A message is sent back to Chloe and the Data Team, and the bugfix is pushed to production.

Lessons Learned

What went well
- Fast response time
- Upon clear evidence, fast mitigation
What went badly
- Issue went unnoticed for more than a month
- No automatic alert detected it
- Impacted critical reporting such as Main KPIs (Business Targets)
- Chloe already raised the possibility of a bug one day in advance and was not deeply investigated
Where we were lucky
- Impact was localised mostly in June despite the bug being in the code for several weeks

Action Items

Clarify VOIDED status treatment with Finance

7.7 KiB Raw Blame History Unescape Escape