81 lines
No EOL
4.4 KiB
Markdown
81 lines
No EOL
4.4 KiB
Markdown
# 20250304-01 - Verification Bulk Update
|
||
|
||
Managed by Uri (Data side)
|
||
|
||
## Summary
|
||
|
||
- While trying to fix a support ticket in the backend on the 3rd March 2025, Tech team accidentally bulk updated the field Value of the Verification table, adding a concatenation of “Payment Validation: Waiver”.
|
||
- This has had impact on the Power BI reports that depend directly or indirectly on this field, namely: Bookings report in SH Reporting and Guest Satisfaction on Guest Insights.
|
||
- After a restore carried out by Tech team and a refresh carried out by Data team, the issue has been fully fixed on the morning of 5th March 2025.
|
||
|
||
## Impact
|
||
|
||
**On Data side:**
|
||
|
||
**Superhog Reporting (production)**
|
||
|
||
- **Report**: Bookings
|
||
- **Tabs**: All
|
||
- **Impact**: Filters of Chosen Fee/Waiver/CheckInCover/NoCover do not show the correct information.
|
||
|
||
**Guest Insights**
|
||
|
||
- **Report**: Guest Satisfaction
|
||
- **Tab**: Only Guest Responses
|
||
- **Impact**: The value shown on Selected Payment Option is wrong
|
||
|
||
The rest of Power BI Reports have been working normally.
|
||
|
||
**On Finance side:**
|
||
|
||
Delay of ~1 day on the invoicing process that depends on SH reporting - Bookings.
|
||
|
||
## Timeline
|
||
|
||
| Time (UTC) | Event |
|
||
| --- | --- |
|
||
| 2025-03-03 13:07 | At 13:07h on 3rd March 2025, a bulk update on the Verification table, on the field Value is done in the Backend. This effectively concatenates any existing verification value with Payment Validation: Waiver. This was unintentional with the actual goal being updating a single record manually. |
|
||
| 2025-03-04 06:00 | DWH scheduled run happens. This captures the changes as Updated Date was also modified. Thus reports depending on this Verification value will be affected. No data alert is raised in this regard, thus going unnoticed. |
|
||
| 2025-03-04 12:40 | Gus and Lawrence communicate to Uri about the issue. Also, Gus posts a message in #all-staff. Uri checks impacts on Data side. |
|
||
| 2025-03-04 13:36 | Uri posts a message in #data about the impacts. Mostly, SH reporting production on Booking tabs have issues on the Chosen Fee/Waiver/CheckInCover/NoCover. Additionally, Guest Satisfaction shows wrong values on the Selected Payment Option. |
|
||
| 2025-03-04 17:50 | Lawrence confirms to Uri that the backend data has been restored normally. |
|
||
| 2025-03-05 7:05 | Uri manually triggers the refresh of the Verification table in Airbyte. |
|
||
| 2025-03-05 7:34 | Uri manually triggers the full-refresh on DBT for the model stg_core__verification in production. |
|
||
| 2025-03-05 7:37 | Uri manually triggers the usual DBT run of all models. |
|
||
| 2025-03-05 7:55 | DWH and Power BI updated. Communication sent in #data to notify users that the incident is now resolved. |
|
||
|
||
## Root Cause(s)
|
||
|
||
On the 3rd March 2025, Tech team was asked to back fill a payment that had been recorded in Stripe but not recorded in the Backend database.
|
||
|
||
This involved:
|
||
|
||
- Inserting one record in the Payment table
|
||
- Inserting one record in the VerificationToPayment table
|
||
- Updating the Value field in one record in the Verification table to append the description of the payment
|
||
|
||
Unfortunately, the where clause of the update was incorrect and reviews missed it. So accidently the update appended the payment description to every record in the Verification table along with the updated date.
|
||
|
||
The value appended was 'Payment Validation: Waiver’. No other critical data had been affected.
|
||
|
||
## Resolution and recovery
|
||
|
||
- From Tech side:
|
||
- Updating the Value and UpdatedDate in the live data with the values from the backup data for any records, where the updated date matches the date of the change.
|
||
- Manual check the value of any records updated since the change to assess what changes need to be made, if any.
|
||
- From Data side:
|
||
- Manual refresh of the Verification table once Tech resolution has been in place and general re-run.
|
||
|
||
## **Lessons Learned**
|
||
|
||
- What went well
|
||
- Quick communication from Gus and Lawrence, knowing that this issue could have impacted Data Reporting.
|
||
- In-detail communication from Lawrence that helped identifying the impacts and course of action from Data side quickly.
|
||
- What went badly
|
||
- No data alert was raised - Data team was unaware of the issue until Tech team communicated it.
|
||
- Where we were lucky
|
||
- Considering the potential risk of a bulk update, an update on the Value of the Verification table is not impacting massively on Data side since we rely mostly on other fields. Thus the impact has been limited, but not zero.
|
||
|
||
## Action Items
|
||
|
||
## Appendix |