Commit graph

799 commits

Author SHA1 Message Date
Joaquin Ossa
0cb03e0808 Merged PR 2707: E-Deposit users to staging
# Description

E-Deposit users to staging to have currencies for PBI report

# Checklist

- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [x] I have checked for DRY opportunities with other models and docs.
- [x] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #20125
2024-09-03 12:53:41 +00:00
Joaquin Ossa
322b122925 Added missing tests 2024-09-03 10:21:23 +02:00
Joaquin Ossa
fa6f7f8ff8 Filtering out test users so all tests work correctly 2024-09-03 10:15:20 +02:00
Joaquin Ossa
ccef428020 Merged PR 2694: Basic model changes for edeposit
# Description

Basic model changes for edeposit

# Checklist

- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [x] I have checked for DRY opportunities with other models and docs.
- [x] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #20125
2024-09-02 15:01:40 +00:00
Joaquin Ossa
89792cf0b7 final comments 2024-09-02 17:01:18 +02:00
Joaquin Ossa
6dedbc04d7 Added more tests but still waiting confirmation on tests users from Ray and Ana 2024-09-02 16:55:56 +02:00
Joaquin Ossa
ffef9e3ff2 Added all date_utc fields 2024-09-02 12:52:16 +02:00
Joaquin Ossa
129b00e29b Added more description and tests to schema 2024-09-02 12:44:05 +02:00
Joaquin Ossa
ee4d213274 edeposit_users to staging 2024-09-02 11:33:25 +02:00
Joaquin Ossa
b6a752fd74 edeposit_users to staging 2024-09-02 11:33:21 +02:00
Joaquin Ossa
46d5e7c3c5 Updating with Ray's comments 2024-09-02 11:16:51 +02:00
Oriol Roqué Paniagua
d8e6ee3ab0 Merged PR 2704: Adding New Dash exposures
# Description

Adds New Dash exposures to close the ticket

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs. **N/A**
- [ ] I've picked the right materialization for the affected models. **N/A**

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Adding New Dash exposures

Related work items: #19570
2024-09-02 07:22:29 +00:00
Pablo Martín
dad16f28b9 Merged PR 2690: VAT rates for guest services seed model
# Description

Adds a seed model with the VAT rates by country for guest services, as provided by Finance team.

This is a necessary step towards calculating taxes in the intermediate layer.

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #20043
2024-08-30 10:49:30 +00:00
Joaquin Ossa
7a77691b89 deleted schema from edeposit reporting 2024-08-30 10:58:24 +02:00
Joaquin Ossa
fd98f31fdd Kept basic models to reduce complexity of
models/20125_edeposit_migration_agg_model_reporting
2024-08-30 10:54:00 +02:00
Joaquin Ossa
42510bbb4d Just committing to save change and create a new branch for basic to push basic changes 2024-08-30 10:33:43 +02:00
Pablo Martin
a839aa4eb3 fix typo 2024-08-29 17:29:21 +02:00
Oriol Roqué Paniagua
7ba65999c3 Merged PR 2687: Materialize int_core__verification_payments as a table
# Description

Just materializes `int_core__verification_payments` as a table instead as a view to enhance compute.

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them. **Technically I get errors in local but from what I see it's because I have different dumps for the currency conversion and the other sources. There's no such cases in prod from what I observed.**
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #19082
2024-08-29 14:25:18 +00:00
Joaquin Ossa
6adc424963 addressed Pablo's comments, removed the repetitive casts, added some nut_null tests and fixed some of the names and descriptions discrepancies 2024-08-29 14:25:00 +02:00
Joaquin Ossa
ad2eb2544c edeposit_agg_fee_per_user to reporting 2024-08-29 11:09:09 +02:00
Joaquin Ossa
951bc70123 Merged PR 2671: New aggregated model for E-deposit report
# Description

New aggregated model for E-deposit report
@<Oriol Roqué Paniagua> not sure if this is what you had in mind with categorizing the cases in a variable, if not let me know so maybe we can check it together

# Checklist

- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [x] I have checked for DRY opportunities with other models and docs.
- [x] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #20125
2024-08-29 08:33:45 +00:00
Oriol Roqué Paniagua
2f77c8eea8 Merged PR 2676: Propagates Billing Country information
# Description

Propagates Billing Country information in unified_user and user_host intermediate models. This is a necessary step towards providing KPIs segmented by Billing Country.

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #19082
2024-08-29 08:25:05 +00:00
Joaquin Ossa
189e77dd76 fixed variable definitions and added comments for currency-less fees 2024-08-29 09:30:36 +02:00
Pablo Martin
60ae3f2844 wip, but tests are not picked up 2024-08-28 17:15:33 +02:00
Joaquin Ossa
be59ab258a Fixed test 2024-08-28 16:42:10 +02:00
Pablo Martín
0137179396 Merged PR 2669: Scheduled tests
# Description

This PR provides a suggested script to run `dbt test` in production, leave logs and send slack messages upon success/failure. It also updates the readme details on how to deploy so that deployers know how to set it.

# Checklist

- __NA__ ~~[ ] The edited models and dependants run properly with production data.~~
- __NA__ ~~[ ] The edited models are sufficiently documented.~~
- __NA__ ~~[ ] The edited models contain PK tests, and I've ran and passed them.~~
- __NA__ ~~[ ] I have checked for DRY opportunities with other models and docs.~~
- __NA__ ~~[ ] I've picked the right materialization for the affected models.~~

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #16959
2024-08-28 14:40:41 +00:00
Joaquin Ossa
0b6239e5c2 New aggregated model in intermediate for e-deposit report 2024-08-28 16:38:30 +02:00
Pablo Martin
10aace0d3f fix log file path 2024-08-28 16:21:07 +02:00
Pablo Martin
2481863536 update readme 2024-08-28 16:20:51 +02:00
Joaquin Ossa
958dc97ac2 Merged PR 2656: e-deposit verifications data to intermediate
# Description

e-deposit verifications data to intermediate
This holds data for each user of the amount of revenue generated each month of creation and check out month

# Checklist

- [X] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [x] I have checked for DRY opportunities with other models and docs.
- [x] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #20125
2024-08-28 13:51:07 +00:00
Joaquin Ossa
b333b45891 Added some comments to make it clear that ids here are unrelated to core dwh, I will come back to modify the schemas when Ray answers all of our questions related to this data 2024-08-28 15:22:17 +02:00
Joaquin Ossa
167645428e renamed columns 2024-08-28 15:22:17 +02:00
Joaquin Ossa
5892fe7cbb Changed it to basic model 2024-08-28 15:21:08 +02:00
Joaquin Ossa
a4dc798f86 e-deposit model WIP 2024-08-28 15:15:27 +02:00
Joaquin Ossa
a1e3174700 e-deposit verifications data to intermediate 2024-08-28 15:15:27 +02:00
Joaquin Ossa
7786f2e770 1st commit edeposit_verifications 2024-08-28 15:12:05 +02:00
Oriol Roqué Paniagua
2facaceec0 Merged PR 2662: New Dash MVP - with. fix on production issue
# Description

Same PR as before, just adds a new commit that fixes my silly issue in prod. I owe some drinks :D

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #19570
2024-08-28 10:09:08 +00:00
Joaquin Ossa
f6f4ca83b7 Merged PR 2658: fixed testing for core__check_in_cover_listings
# Description

fixed testing for core__check_in_cover_listings and removed null in accepted values for int_core__bookings in verification_request_booking_source

# Checklist

- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [x] I have checked for DRY opportunities with other models and docs.
- [x] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #20229
2024-08-28 10:07:11 +00:00
Oriol Roqué Paniagua
a0b5b0bda1 Merged PR 2660: Revert 'Automating new dash query'
# Description

_Describe your motivation and changes here._

# Checklist

- [ ] The edited models and dependants run properly with production data.
- [ ] The edited models are sufficiently documented.
- [ ] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

# Description

Automates the new dash query through DWH. It's a first step towards enabling a very simple PBI reporting on new dash performance. Data is aggregated at user level with the main performance indicators that we're currently reporting. I added a few more informative fields from user side.

Changes:

* in intermediate, creation of `int_core__new_dash_user_overview`. It mainly uses the models in previous PRs to avoid crazy computations, since most of the logic has already been implemented. The only thing is that it's "hardcoded" (with variable) the categorisation of what is NOT a paid service.
* in reporting, creation of `core__new_dash_user_overview`. It just exposes the info from intermediate to reporting.
* added schema entries for intermediate/core and reporting/core, it's the same description for both cases.

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Reverts !2657

Related work items: #19570
2024-08-28 09:40:09 +00:00
Oriol Roqué Paniagua
8f38e94a5c Merged PR 2657: Automating new dash query
# Description

Automates the new dash query through DWH. It's a first step towards enabling a very simple PBI reporting on new dash performance. Data is aggregated at user level with the main performance indicators that we're currently reporting. I added a few more informative fields from user side.

Changes:

* in intermediate, creation of `int_core__new_dash_user_overview`. It mainly uses the models in previous PRs to avoid crazy computations, since most of the logic has already been implemented. The only thing is that it's "hardcoded" (with variable) the categorisation of what is NOT a paid service.
* in reporting, creation of `core__new_dash_user_overview`. It just exposes the info from intermediate to reporting.
* added schema entries for intermediate/core and reporting/core, it's the same description for both cases.

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #19570
2024-08-28 09:32:17 +00:00
Joaquin Ossa
46a6ff057d fixed testing for core__check_in_cover_listings 2024-08-28 10:47:31 +02:00
Oriol Roqué Paniagua
00b8c66709 Merged PR 2652: Adds accommodation to product bundle in intermediate
# Description

Adds accommodation to product bundle in intermediate. The source table is currently empty, because there's no product bundle different to Basic Screening applied into a listing yet.
Thus, it's possible that this modelisation needs to change in the future since not having data forces us to trust the business logic without a proper technical documentation.

# Checklist

- [ ] The edited models and dependants run properly with production data. **N/A no data**
- [X] The edited models are sufficiently documented.
- [ ] The edited models contain PK tests, and I've ran and passed them. **N/A no data**
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models. **N/A no data - but view should make the trick for the time being**

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #19570
2024-08-27 14:05:24 +00:00
Joaquin Ossa
67468a3220 Merged PR 2653: fixed bookings test to include null for verification_request_booking_source
# Description

Fixed bookings test to include null for verification_request_booking_source

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #20231
2024-08-27 13:48:52 +00:00
Joaquin Ossa
87eac12281 fixed bookings test to include null for verification_request_booking_source 2024-08-27 15:45:22 +02:00
Oriol Roqué Paniagua
b927635cf1 Merged PR 2650: Added int_core__booking_to_product_bundle in intermediate
# Description

Creates a new view with the bookings that come from New Dash, linked to the Product Bundles. Extensive documentation added, please check it out and if something is not clear I'll modify it.

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models. To be discussed. There's not huge data still so view looks ok to me. I guess, maybe, in the future, we will have a flag or similar in int_core__bookings to detect if a bookings comes from New Dash/New Pricing or not, but it's still early to decide. In any case this model is needed for immediate New Dash MVP reporting so I propose to keep it like this as a view and afterwards we can decide if it makes more sense to materialise it differently.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #19570
2024-08-27 12:51:55 +00:00
Oriol Roqué Paniagua
c613d2859f Merged PR 2648: Fixes int_dates_by_deal tests
# Description

Fixes int_dates_by_deal tests

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #20318, #20319
2024-08-27 09:34:07 +00:00
Pablo Martín
3c73bcf965 Merged PR 2646: Add missing edeposit columns
# Description

Adds three more columns to the staging model of `verifications` for edeposit: `ListingId`, `NightlyFee` and `CompanyName`.

# Checklist

- [X] The edited models and dependants run properly with production data.
- ~~[ ] The edited models are sufficiently documented.~~ __No, but I promise they will be.__
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [X] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #20123
2024-08-27 09:22:52 +00:00
Oriol Roqué Paniagua
cb3d1e628b Merged PR 2647: Adding accommodation_to_product_bundle to stg
# Description

Adds accommodation to product bundle table from sync_core to staging. Note that this table still has no data because so far no listing has an associated product bundle.

Small change: removed in schema pending confirmation comments from Lou now that we've got an answer confirming it.

# Checklist

- [ ] The edited models and dependants run properly with production data: **N/A - there's no data :(**
- [X] The edited models are sufficiently documented.
- [ ] The edited models contain PK tests, and I've ran and passed them. **N/A - there's no data :(**
- [X] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models. **N/A - there's no data :(**

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #19570
2024-08-27 08:57:23 +00:00
Pablo Martin
4cc456c3b0 add missing columns to model 2024-08-27 10:33:18 +02:00
Pablo Martin
9432313adc pending todo so i don't forget 2024-08-26 16:30:18 +02:00