sh-notion/notion_data_team_no_files/20250124-01 - Booking invoicing incident 1880446ff9c9803fb830f8de24d97ebb.md

# 20250124-01 - Booking invoicing incident

# Booking invoicing incident

Managed by: Uri

## Summary

- Components involved: `sh-invoicing-exporter` tool
- Started at: 2024-11-29 14:33 UTC
- Detected at: 2025-01-24 10:06 UTC
- Mitigated at: 2025-01-28 16:06 UTC
- Resolved at: 2025-02-05 14:52 UTC (after post-mortem)

While working on the analysis on Booking Fees per Billable Booking Decrease, Uri noticed that two of the main clients in terms of Booking Fees were not invoiced for November and December 2024. After confirmation from Finance side and deep-dive, we noticed that something was odd with the exports generated by sh-invoicing-exporter tool.

The reason is a faulty modification on the sh-invoicing-exporter tool after the latest invoicing incident on November 4th, 2024. While the change properly handles the agreed logic, it undesirably introduces an error for invoicing clients that have the billing set to Verification Start Date but do not have Verification Requests associated to their bookings. This issue did not happen prior to November’s modification.

**A total of £20,777.83 was not invoiced** in the combined months of November and December 2024. The remediation carried out on January 28th 2025 accounted for a **late invoicing of 99.1%** of the missed revenue, pending payment.

## Impact

- On the invoicing process:
    - November and December invoices process were carried out in apparent normality while we avoided invoicing some clients.
        - A first estimate puts the missing revenue around 10k to 15k GBP per month missed in terms of Booking Fees. The estimated impact over 2 months would be around 20k to 30k GBP.
        - **The final figure after the invoicing process has been re-run on January 28th 2025 is:**
            - **Combined over November and December - Revenue £20,777.83**
            - **Total Invoiced (late) - Revenue £20,592.76**
            - **Total lost revenue - £185.07**
            
            Lost Revenue comes from clients that have churned in the period (November & December) of which we have no anticipation of receiving the funds, therefore have not been invoiced.
            

## Timeline

All times are UTC, in 00-24h format.

| Time | Event |
| --- | --- |
| 2024-11-29 14:33 | A new fix to handle [November invoicing incident](20241104-01%20-%20Booking%20invoicing%20incident%20due%20to%20bu%2082f0fde01b83440e8b2d2bd6839d7c77.md) is merged into sh-invoicing-tool. The relevant commit is [6ea412a9320c621560dd780a7cf49617ffdefa8e](https://guardhog.visualstudio.com/Data/_git/data-invoicing-exporter/commit/6ea412a9320c621560dd780a7cf49617ffdefa8e?refName=refs%2Fheads%2Fmain). |
| 2024-12-02 (approx.) | Pablo runs sh-invoicing-exporter as usual, and provides the Exports to Finance team. These do not contain information to bill some clients, but remains unnoticed. This is the first time the process runs with the latest modifications described [here](https://www.notion.so/Fixing-the-invoicing-incident-13d0446ff9c98056a65bc3676a34873c?pvs=21), after November 2024 invoicing incident. |
| 2025-01-02 (approx.) | Uri runs sh-invoicing-exporter as usual, and provides the Exports to Finance team. These do not contain information to bill some clients, but remains unnoticed. |
| 2025-01-21 13:49 | Matt reports to Data team that the Booking Fees revenue is low in December, to be investigated. |
| 2025-01-22 16:00 (approx.) | Uri starts analysing the decrease of Booking Fees per Billable Booking, as requested by Matt. |
| Friday 2025-01-24 10:06 | Uri raises that two clients, 20529225110 - Lavanda (UK) and 6030475449 - Hospitable Inc (US) have not been invoiced nor in November 2024 nor in December 2024. Asks Finance for confirmation from their POV. |
| 2025-01-24 12:03 | After a sync between Nathan and Uri, it’s confirmed that Lavanda was not invoiced. The issue seems to be related to the invoicing exports delivered by Data team to Finance team, on sh-invoicing-tool. On trying to re-create the exports for Lavanda, we got empty files (no billable bookings and no booking fees) for December, November and October. The fact that Lavanda was actually invoiced correctly in October but Uri couldn’t replicate the invoice export for October points to a potential issue either on sh-invoicing-tool or the backend data. |
| 2025-01-24 12:19 | Uri confirms we have the same problem with Hospitable. However, Uri is able to properly create the invoicing export for Home Team Vacation Rentals LLC, a client that was actually invoiced. At this stage it seems the issue might come from changes introduced in sh-invoicing-tool, and the hypothesis of backend data issues loses support.
Ben R. is added in the loop for visibility on a potential invoicing incident, while Uri further investigates on his own. |
| 2025-01-24 13:16 | Uri confirms this is an incident after checking the sh-invoicing-exporter tool, localising the [faulty commit](https://guardhog.visualstudio.com/Data/_git/data-invoicing-exporter/commit/6ea412a9320c621560dd780a7cf49617ffdefa8e?refName=refs%2Fheads%2Fmain) and reproducing the no-results for Lavanda manually. It’s also confirmed that the issue is on sh-invoicing-tool. An incident channel is created. |
| 2025-01-24 13:37 | Uri provides the explanation of the root cause of the incident in the channel, specifically:

- The issue lies in [data-invoicing-exporter](https://guardhog.visualstudio.com/Data/_git/data-invoicing-exporter). This is the tool that we use from Data side to generate the Excel files to support old dash invoicing.
- This is a consequence, probably undesired, from the changes carried out after the latest incident on November 4th. I'm referring to the incident described [in this Notion](20241104-01%20-%20Booking%20invoicing%20incident%20due%20to%20bu%2082f0fde01b83440e8b2d2bd6839d7c77.md), the remediation explained [in this other Notion](https://www.notion.so/Fixing-the-invoicing-incident-13d0446ff9c98056a65bc3676a34873c?pvs=21) and that was discussed in this channel #invoicing-firefighting
- The impact from my current understanding is that any Host that would have billing set to `VerificationStartDate` but does NOT have Guest Journeys associated to the bookings, will not have been billed from November onwards. This is exactly the case for Lavanda, for instance. This can happen again in the next invoicing cycle if not fixed.

In detail:
- This is the [faulty commit](https://guardhog.visualstudio.com/Data/_git/data-invoicing-exporter/commit/6ea412a9320c621560dd780a7cf49617ffdefa8e?refName=refs%2Fheads%2Fmain) . Specifically, the change breaks the logic on `VerificationRequestId IS NULL`, which exactly handles the Lavanda situation of having Bookings but not Guest Journeys (note that this was covered before, while not after the merge). This explains why I was unable to replicate the extraction of October, even though it worked back in the days, because I was using today the latest version of the tool.
- In essence, the new logic assumes it can compare the `LinkUsed` from `VerificationRequest` table to `JoinDate` from `SuperhogUser` table. For Lavanda case, the issue is that `LinkUsed` will always be null and thus the condition is always false, excluding ALL bookings. |
| 2025-01-24 13:45 | Uri proposes a first solution to handle billing method set to `VerificationStartDate`
- If the Booking has a Guest Journey (Verification Request) associated with, we keep the latest logic. This means, we keep using the logic implemented after November 2024 that is currently in place, by properly handling the new/returning guests
- If the Booking has NOT a Guest Journey (Verification Request) associated with, we keep the previous logic. This means, we will charge the booking if the Guest `JoinDate` happened within the month

This effectively recreates around 1.500 bookings for Lavanda in October, while we had 1.489 actually billed in October |
| 2025-01-24 14:45 | Missing Booking Fees impact is estimated to be mostly coming from Lavanda and Hospitable (~88% in total). Some smaller accounts exist, part of those have already churned. This makes an estimate figure of 10k-15k GBP missed per month, that could be 20k-30k GBP in the 2 months.  |
| 2025-01-24 14:47 | Nathan suggests to re-do the exports for the missing clients identified in the previous estimation to get the actual figure of impact. Pending validation of logic from Ben R. before moving forward. |
| 2025-01-24 15:22 | Uri creates a code PR in sh-invoicing-tool for Ben R. to review, with the suggested changes. This is the PR: [Ensure Billing of Bookings without Verification Requests](https://guardhog.visualstudio.com/Data/_git/data-invoicing-exporter/pullrequest/4185) |
| 2025-01-24 15:49 | Ben and Uri discuss the business logic of the change, lacking understanding of why verification request would be null for those bookings. While this is actually the case, there’s a possibility that it’s linked to KYG bookings. In short, while the suggested change could properly invoice the missing accounts, it’s possible that this can ultimately generate wrong invoices for other clients that are not supposed to be invoiced. Since it’s quite confusing at the moment, Ben and Uri agree to take a look at this on Monday. A message is sent to the invoicing channel to clarify that we won’t have the exports by Friday afternoon since we need to deep-dive into the code. |
| 2025-01-24 15:50 | Nathan acknowledges the proposed plan and comments that he’d chat with Leo to notify that we’d be preparing the missing invoices for the 2 main clients, Lavanda and Hospitable, subject to tech work.
End of the incident management for Friday 24th January 2025. |
| Monday 2025-01-27 11:52 | Ben and Uri sync. It is agreed that the initially proposed fix is ok to handle the cases of Bookings without Verification Requests, likely because these are coming from APIs. A follow up message is sent to proceed with the implementation and re-create November and December exports.
In parallel, Ben and Uri discuss on New Dash exclusion for the old invoicing tool (not related to the incident). Need clarification from Finance side to proceed on New Dash regard. |
| 2025-01-27 13:47 | Uri merges [the fix](https://guardhog.visualstudio.com/Data/_git/data-invoicing-exporter/commit/2e464aa9bc420540c9536ed500a0af15b9b1e3de?refName=refs%2Fheads%2Fmain) and updates the sh-invoicing-tool in Airbyte VM to version 3.1.1. A run for November 2024 with the 2 main affected accounts (Lavanda and Hospitable) is successful in the sense that now gathers Booking Fees.  |
| 2025-01-27 14:17 | Uri triggers the full run for November 2024 on version 3.1.1. |
| 2025-01-27 17:05 | Full run for November 2024 is finished. After checking the files, it’s clear that there has been an error somewhere as we’re observing a 10X difference vs. what was observed for January export. |
| 2025-01-27 18:41 | Following the case of Home Team Vacation Rentals, it is observed that the issue comes from the fact that we’re missing some parenthesis in the fixed code. A rookie mistake but at least at this stage it does not seem to be something that would challenge the proposed business logic for the fix.  |
| 2025-01-27 18:55 | Uri merges [the fix](https://guardhog.visualstudio.com/Data/_git/data-invoicing-exporter/commit/612d991190bca93dd662ace790c866d5d10307dc?refName=refs%2Fheads%2Fmain) and updates the sh-invoicing-tool in Airbyte VM to version 3.1.2. First tests on Lavanda, Hospitable and Home Team Vacation Rentals are successful. |
| 2025-01-28 07:55 | Uri triggers the full run for November 2024 on version 3.1.2. |
| 2025-01-28 09:47 | Full run for November 2024 is finished. At first glance, it looks ok. File is shared with Nathan |
| 2025-01-28 10:06 | Uri triggers the full run for December 2024 on version 3.1.2. |
| 2025-01-28 11:14 | Nathan confirms November files look good and that Finance will start raising them |
| 2025-01-28 12:54 | Full run for December 2024 is finished. At first glance, it looks ok. File is shared with Nathan |
| 2025-01-28 15:28 | Nathan finishes the missing invoices. Total invoiced revenue (late) accounts for 20,592.76 GBP over an expected 20,777.83 GBP. The difference -185,07 GBP is lost revenue due to Clients that churned in the period (Nov & Dec 2024) of which we have not anticipation of receiving the funds, therefore have not been invoiced. Expected payment received in the next week or 2. |
| 2025-01-28 16:06 | Incident is mitigated.  |

## Root Cause(s)

After the last invoicing incident, available here:

[20241104-01 - Booking invoicing incident due to bulk UpdatedDate change](20241104-01%20-%20Booking%20invoicing%20incident%20due%20to%20bu%2082f0fde01b83440e8b2d2bd6839d7c77.md) 

there has been agreement on how to avoid dependency on the UpdatedDate field for price plans that are supposed to be charged on Verification Start, which has proven not to be reliable for invoicing purposes.

The agreement on the new logic for invoicing is described here:

[Decisions](https://www.notion.so/Decisions-13d0446ff9c980248babfd2b83b7781c?pvs=21) 

The changes were implemented in sh-invoicing-exporter as part of the [new version 3.1.0](https://guardhog.visualstudio.com/Data/_git/data-invoicing-exporter/commit/6ea412a9320c621560dd780a7cf49617ffdefa8e?refName=refs%2Fheads%2Fmain), and these are in line with the decisions taken for November incident. However, this new logic does not contemplate the Billing of Bookings that do not have a Verification Requests linked to them, while it was considered prior to version 3.1.0. This lead to avoid billing Bookings without Verification Requests for the months of November 2024 and December 2024.

## Resolution and recovery

The resolution considers a mix of the logic prior and after November 2024 incident. Namely:

- For Old Dashboard clients who has active price plans in period that are charged according to the Verification Start:
    - If Booking has a Verification Request linked to it:
        - If a Guest is New, then:
            - Charge if Guest Join Date is within the invoicing period
        - If a Guest is Returning, then:
            - Charge if Link Used Date is within the invoicing period
    - If Booking does not have a Verification Request linked to it:
        - Charge if Guest Join Date is within the invoicing period

Coding wise, this just means treating the possible Link Used Date being null and, if so, apply the same condition as Guest is New. 

This change was first implemented on 27th January 2025 under [version 3.1.1](https://guardhog.visualstudio.com/Data/_git/data-invoicing-exporter/commit/2e464aa9bc420540c9536ed500a0af15b9b1e3de?refName=refs%2Fheads%2Fmain), followed by a fix after an unintended bug. The last working version corresponds to [version 3.1.2](https://guardhog.visualstudio.com/Data/_git/data-invoicing-exporter/commit/612d991190bca93dd662ace790c866d5d10307dc?refName=refs%2Fheads%2Fmain), also from 27th January 2025. 

The invoicing reports for November 2024 and December 2024 were recreated under the fixed version 3.1.2 during the morning of 28th January 2025 and shared with Finance to act upon.

During the afternoon of 28th January 2025, Finance (Nathan) posted the remaining invoices to account for a total of £20,592.76 of late revenue**.**

## **Lessons Learned**

*List of knowledge acquired. Typically structured as: What went well, what went badly, where did we get lucky*

- What went well
    - Relative consistent figures on Power BI & DWH KPIs vs. Finance’s P&L, which allowed for fast deep-dive
    - Fast and effective cross-department collaboration once the issue was initially flagged
    - Well documented incident management on November 2024 allowed for a quick and easy understanding on what was going on during January 2025 incident
- What went badly
    - Being unable to identify an issue in the critical area of invoicing for more than 2 months
    - Being unable to notice for 2 months that two of the biggest clients were not invoiced
- Where did we get lucky
    - Prioritising analysis on Booking Fees per Billable Booking decrease after being one of the newly defined financial levers, which lead to discovering the invoicing issue.
    - The incident was reported a week before the end of January, thus allowing for recovery time before extending the impact yet to a third invoicing cycle

## Action Items

- [x]  Xero automation in Power BI - Global MoM revenue aggregations evolution
- [x]  Xero automation in Power BI - Top X clients MoM revenue aggregations evolution
- [ ]  Further automation improvements - Xero (for source of truth in actual invoiced amount) + Hubspot (for churn/onboarding/AM info) + Backend (for what we should have invoiced)
- [ ]  New Dash “Chargeable” services - To check with Ben and Nathan to ensure logic is consistent

## Appendix

*Miscellanea corner for anything else you might want to include*

-