counterweight/data-dwh-dbt-project
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merged PR 2832: Format YAML on save + complements

Browse source 
# Description

This PR:
- Adds some `vscode` settings that make VSCode autoformat yaml files on save, like we do with `.sql` files. This should help with formatting consistency.
- I also made some notes about this on the readme.
- Finally, I made a couple of tiny unrelated changes on the readme since I was there.

Related work items: #21344
This commit is contained in:
Pablo Martín 2024-09-13 08:33:26 +00:00
commit ac1aed029d
15 changed files with 477 additions and 567 deletions
Download patch file Download diff file Expand all files Collapse all files

3
.vscode/settings.json vendored
View file

@ -12,6 +12,9 @@
}, },
"dbt.enableNewLineagePanel": true, "dbt.enableNewLineagePanel": true,
"dbt.enableNewDocsPanel": true, "dbt.enableNewDocsPanel": true,
"[yaml]": {
"editor.formatOnSave": true
},
"yaml.schemas": { "yaml.schemas": {
"https://raw.githubusercontent.com/dbt-labs/dbt-jsonschema/main/schemas/dbt_yml_files.json": [ "https://raw.githubusercontent.com/dbt-labs/dbt-jsonschema/main/schemas/dbt_yml_files.json": [
"/**/*.yml", "/**/*.yml",

11
README.md
View file

@ -51,9 +51,11 @@ pg_restore -h localhost -U postgres -W -d dwh xero.dump
This repo works in a trunk-based-development philosophy (<https://trunkbaseddevelopment.com/>). This repo works in a trunk-based-development philosophy (<https://trunkbaseddevelopment.com/>).
When working on Data modeling stuff (models, sources, seeds, docs, etc.) use a `models` branch (i.e. `models/churned-users`). It's fine and encouraged to build incrementally towards a `reporting` level table with multiple PRs as long as you keep the model buildable along the way. If your branch is related to a work item from DevOps, we encourage adding the ticket number in the branch name. For example: `models/123-some-fancy-name`. If you don't have a ticket number, you can simply do a `NOTICKET` one: `models/NOTICKET-some-fancy-name`.
For other matters, use a `chores` branch (i.e. `chores/add-dbt-package`). When working on Data modeling stuff (models, sources, seeds, docs, etc.) use a `models` branch (i.e. `models/782-churned-users`). It's fine and encouraged to build incrementally towards a `reporting` level table with multiple PRs as long as you keep the model buildable along the way.
For other matters, use a `chores` branch (i.e. `chores/656-add-dbt-package`).
## Project organization ## Project organization
@ -89,6 +91,9 @@ We organize models in three folders:
- Split schema and domain with double underscode (ie `stg_core__booking`). - Split schema and domain with double underscode (ie `stg_core__booking`).
- Always use sources to read into staging models. - Always use sources to read into staging models.
- SQL formatting should be done with `sqlfmt`. - SQL formatting should be done with `sqlfmt`.
- YAML files:
- Should use the `.yml` extension, not `.yaml`.
- Should be autoformatted on save. If you install [this vscode extension](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml), autoformatting should happen out of the box thanks to the settings included in the `.vscode/settings.json` file.
- Other conventions - Other conventions
- In staging, enforce a `lower()` to user UUID fields to avoid nasty propagations in the DWH. - In staging, enforce a `lower()` to user UUID fields to avoid nasty propagations in the DWH.
@ -131,6 +136,4 @@ To maintain:
## Stuff that we haven't done but we would like to ## Stuff that we haven't done but we would like to
- Automate formatting with git pre-commit. - Automate formatting with git pre-commit.
- Define conventions on testing (and enforce them).
- Define conventions on documentation (and enforce them).
- Prepare a quick way to replicate parts of the `prd` dwh in our local machines. - Prepare a quick way to replicate parts of the `prd` dwh in our local machines.

276
models/intermediate/core/schema.yaml → models/intermediate/core/schema.yml
View file

@ -177,8 +177,7 @@ models:
- unique - unique
- name: int_core__vr_check_in_cover - name: int_core__vr_check_in_cover
description: description: "This tables holds information on verification requests
"This tables holds information on verification requests
with Ckeck-in Hero available for the guests." with Ckeck-in Hero available for the guests."
columns: columns:
- name: id_verification_request - name: id_verification_request
@ -224,8 +223,7 @@ models:
- name: is_verification_request_complete - name: is_verification_request_complete
data_type: boolean data_type: boolean
description: description: "True if the verification request is considered
"True if the verification request is considered
complete, AKA the guest has finished the full guest journey." complete, AKA the guest has finished the full guest journey."
- name: is_past_check_in - name: is_past_check_in
@ -238,13 +236,11 @@ models:
- name: cover_was_purchased - name: cover_was_purchased
data_type: boolean data_type: boolean
description: description: "Boolean to indicate if the cover was purchased by the guest or not."
"Boolean to indicate if the cover was purchased by the guest or not."
- name: address_validation_passed - name: address_validation_passed
data_type: boolean data_type: boolean
description: description: "Boolean to indicate if the validation was approved or not.
"Boolean to indicate if the validation was approved or not.
In case there was a problem with the validation it will be Null" In case there was a problem with the validation it will be Null"
- name: address_validation_notes - name: address_validation_notes
@ -411,8 +407,7 @@ models:
- name: currency - name: currency
data_type: text data_type: text
description: description: "The currency in which the transaction actually happened."
"The currency in which the transaction actually happened."
- name: amount_in_gbp - name: amount_in_gbp
data_type: numeric data_type: numeric
@ -422,8 +417,7 @@ models:
- name: payment_status - name: payment_status
data_type: character varying data_type: character varying
description: description: "The status of the payment. It can be one of: Paid, Refunded, Refund Failed."
"The status of the payment. It can be one of: Paid, Refunded, Refund Failed."
- name: payment_paid_date_utc - name: payment_paid_date_utc
data_type: date data_type: date
@ -539,7 +533,7 @@ models:
description: The dimension or granularity of the metrics. description: The dimension or granularity of the metrics.
tests: tests:
- accepted_values: - accepted_values:
values: values:
- global - global
- by_number_of_listings - by_number_of_listings
- by_billing_country - by_billing_country
@ -575,7 +569,7 @@ models:
description: The dimension or granularity of the metrics. description: The dimension or granularity of the metrics.
tests: tests:
- accepted_values: - accepted_values:
values: values:
- global - global
- by_number_of_listings - by_number_of_listings
- by_billing_country - by_billing_country
@ -611,7 +605,7 @@ models:
description: The dimension or granularity of the metrics. description: The dimension or granularity of the metrics.
tests: tests:
- accepted_values: - accepted_values:
values: values:
- global - global
- by_number_of_listings - by_number_of_listings
- by_billing_country - by_billing_country
@ -647,7 +641,7 @@ models:
description: The dimension or granularity of the metrics. description: The dimension or granularity of the metrics.
tests: tests:
- accepted_values: - accepted_values:
values: values:
- global - global
- by_number_of_listings - by_number_of_listings
- by_billing_country - by_billing_country
@ -683,7 +677,7 @@ models:
description: The dimension or granularity of the metrics. description: The dimension or granularity of the metrics.
tests: tests:
- accepted_values: - accepted_values:
values: values:
- global - global
- by_number_of_listings - by_number_of_listings
- by_billing_country - by_billing_country
@ -835,7 +829,7 @@ models:
- name: id_verification_to_payment - name: id_verification_to_payment
data_type: bigint data_type: bigint
description: Unique ID for the relation between the payment verification description: Unique ID for the relation between the payment verification
and the payment at hand. and the payment at hand.
tests: tests:
- unique - unique
- not_null - not_null
@ -1214,12 +1208,12 @@ models:
tests: tests:
- not_null - not_null
- accepted_values: - accepted_values:
values: values:
- '0' - "0"
- '01-05' - "01-05"
- '06-20' - "06-20"
- '21-60' - "21-60"
- '61+' - "61+"
- name: accommodations_booked_in_12_months - name: accommodations_booked_in_12_months
data_type: integer data_type: integer
@ -1364,8 +1358,7 @@ models:
- name: check_in_cover_added_date_utc - name: check_in_cover_added_date_utc
data_type: date data_type: date
description: description: Date the user first included check-in cover
Date the user first included check-in cover
- name: billing_town - name: billing_town
data_type: character varying data_type: character varying
@ -1399,28 +1392,25 @@ models:
- name: verification_request_booking_source - name: verification_request_booking_source
data_type: text data_type: text
description: description: Source type of host of the booking, this could be either;
Source type of host of the booking, this could be either;
- PMS - PMS
- OSL - OSL
- API/MANUAL - API/MANUAL
tests: tests:
- not_null - not_null
- accepted_values: - accepted_values:
values: values:
- 'PMS' - "PMS"
- 'OSL' - "OSL"
- 'API/MANUAL' - "API/MANUAL"
- name: experience_rating - name: experience_rating
data_type: bigint data_type: bigint
description: description: Guest rating of their experience with Superhog from 1 to 5
Guest rating of their experience with Superhog from 1 to 5
- name: guest_comments - name: guest_comments
data_type: character varying data_type: character varying
description: description: Guest comments on their experience with Superhog
Guest comments on their experience with Superhog
- name: is_contactable - name: is_contactable
data_type: boolean data_type: boolean
@ -1429,13 +1419,11 @@ models:
- name: created_at_utc - name: created_at_utc
data_type: timestamp without time zone data_type: timestamp without time zone
description: description: Date and time of response creation
Date and time of response creation
- name: updated_at_utc - name: updated_at_utc
data_type: timestamp without time zone data_type: timestamp without time zone
description: description: Date and time of last update of response
Date and time of last update of response
- name: selected_payment_option - name: selected_payment_option
data_type: character varying data_type: character varying
@ -1469,10 +1457,8 @@ models:
description: | description: |
True if guest payed the fee True if guest payed the fee
- name: int_core__verification_request_booking_source - name: int_core__verification_request_booking_source
description: description: This model contains information on verification requests
This model contains information on verification requests
and the category type of host that manages the associated and the category type of host that manages the associated
booking. booking.
For PMS (Property Manager System) we use the id_integration For PMS (Property Manager System) we use the id_integration
@ -1488,8 +1474,7 @@ models:
columns: columns:
- name: id_verification_request - name: id_verification_request
data_type: bigint data_type: bigint
description: description: Id value for the verification request, there can be more
Id value for the verification request, there can be more
than 1 record for each verification request since they can than 1 record for each verification request since they can
be associated to more than 1 booking be associated to more than 1 booking
tests: tests:
@ -1498,28 +1483,25 @@ models:
- name: created_at_utc - name: created_at_utc
data_type: timestamp without time zone data_type: timestamp without time zone
description: description: Date and time of creation of the verification request
Date and time of creation of the verification request
- name: created_date_utc - name: created_date_utc
data_type: date data_type: date
description: description: Date of creation of the verification request
Date of creation of the verification request
- name: verification_request_booking_source - name: verification_request_booking_source
data_type: text data_type: text
description: description: Source type of host of the booking, this could be either;
Source type of host of the booking, this could be either;
- PMS - PMS
- OSL - OSL
- API/MANUAL - API/MANUAL
tests: tests:
- not_null - not_null
- accepted_values: - accepted_values:
values: values:
- 'PMS' - "PMS"
- 'OSL' - "OSL"
- 'API/MANUAL' - "API/MANUAL"
- name: int_core__verification_requests - name: int_core__verification_requests
description: description:
@ -1607,43 +1589,35 @@ models:
- name: verification_estimated_started_at_utc - name: verification_estimated_started_at_utc
data_type: timestamp without time zone data_type: timestamp without time zone
description: description: The estimated date and time at which the guest started the guest journey.
The estimated date and time at which the guest started the guest journey.
- name: verification_estimated_started_date_utc - name: verification_estimated_started_date_utc
data_type: date data_type: date
description: description: The estimated date on which the guest started the guest journey.
The estimated date on which the guest started the guest journey.
- name: verification_estimated_completed_at_utc - name: verification_estimated_completed_at_utc
data_type: timestamp without time zone data_type: timestamp without time zone
description: description: The estimated date and time at which the guest finished the guest journey.
The estimated date and time at which the guest finished the guest journey.
- name: verification_estimated_completed_date_utc - name: verification_estimated_completed_date_utc
data_type: date data_type: date
description: description: The estimated date on which the guest finished the guest journey.
The estimated date on which the guest finished the guest journey.
- name: link_used_at_utc - name: link_used_at_utc
data_type: timestamp without time zone data_type: timestamp without time zone
description: description: The date and time at which the guest used the link for the verification.
The date and time at which the guest used the link for the verification.
- name: link_used_date_utc - name: link_used_date_utc
data_type: date data_type: date
description: description: The date on which the guest used the link for the verification.
The date on which the guest used the link for the verification.
- name: expire_at_utc - name: expire_at_utc
data_type: timestamp without time zone data_type: timestamp without time zone
description: description: The date and time at which the link for the verification expires.
The date and time at which the link for the verification expires.
- name: expire_date_utc - name: expire_date_utc
data_type: date data_type: date
description: description: The date on which the link for the verification expires.
The date on which the link for the verification expires.
- name: is_deleted - name: is_deleted
data_type: boolean data_type: boolean
@ -1661,18 +1635,17 @@ models:
- name: verification_request_booking_source - name: verification_request_booking_source
data_type: text data_type: text
description: description: Source type of host of the booking, this could be either;
Source type of host of the booking, this could be either;
- PMS - PMS
- OSL - OSL
- API/MANUAL - API/MANUAL
tests: tests:
- not_null - not_null
- accepted_values: - accepted_values:
values: values:
- 'PMS' - "PMS"
- 'OSL' - "OSL"
- 'API/MANUAL' - "API/MANUAL"
- name: success_message - name: success_message
data_type: character varying data_type: character varying
@ -1684,8 +1657,7 @@ models:
- name: rejection_reason - name: rejection_reason
data_type: character varying data_type: character varying
description: description: Reason as to why the guest was rejected.
Reason as to why the guest was rejected.
- name: has_switched_to_mobile - name: has_switched_to_mobile
data_type: boolean data_type: boolean
@ -1707,23 +1679,19 @@ models:
- name: created_at_utc - name: created_at_utc
data_type: timestamp without time zone data_type: timestamp without time zone
description: description: The date and time at which the verification process was created.
The date and time at which the verification process was created.
- name: created_date_utc - name: created_date_utc
data_type: date data_type: date
description: description: The date on which the verification process was created.
The date on which the verification process was created.
- name: updated_at_utc - name: updated_at_utc
data_type: timestamp without time zone data_type: timestamp without time zone
description: description: The date and time at which the last update on the entry happened.
The date and time at which the last update on the entry happened.
- name: updated_date_utc - name: updated_date_utc
data_type: date data_type: date
description: description: The date on which the last update on the entry happened.
The date on which the last update on the entry happened.
- name: dwh_extracted_at_utc - name: dwh_extracted_at_utc
data_type: timestamp with time zone data_type: timestamp with time zone
@ -1765,18 +1733,17 @@ models:
- name: verification_request_booking_source - name: verification_request_booking_source
data_type: text data_type: text
description: description: Source type of host of the booking, this could be either;
Source type of host of the booking, this could be either;
- PMS - PMS
- OSL - OSL
- API/MANUAL - API/MANUAL
- null (bookings without verification request) - null (bookings without verification request)
tests: tests:
- accepted_values: - accepted_values:
values: values:
- 'PMS' - "PMS"
- 'OSL' - "OSL"
- 'API/MANUAL' - "API/MANUAL"
- name: id_staging_host_booking - name: id_staging_host_booking
data_type: bigint data_type: bigint
@ -1784,8 +1751,7 @@ models:
- name: is_duplicate_booking - name: is_duplicate_booking
data_type: boolean data_type: boolean
description: description: A flag that identifies whether the booking is a duplicate.
A flag that identifies whether the booking is a duplicate.
A booking is considered a duplicate if there's an older booking with the same user, A booking is considered a duplicate if there's an older booking with the same user,
accomodation and check-in date. If there are two or more bookings with the same user, accomodation and check-in date. If there are two or more bookings with the same user,
@ -1796,8 +1762,7 @@ models:
- name: is_duplicating_booking_with_id - name: is_duplicating_booking_with_id
data_type: bigint data_type: bigint
description: description: "If is_duplicate_booking is True then gives id_booking"
"If is_duplicate_booking is True then gives id_booking"
- name: booking_state - name: booking_state
data_type: character varying data_type: character varying
@ -1892,8 +1857,7 @@ models:
description: "" description: ""
- name: int_core__check_in_cover_listings - name: int_core__check_in_cover_listings
description: description: This model contains information about hosts and their listings
This model contains information about hosts and their listings
that offer check in cover. that offer check in cover.
It has basic information on the users and listings like country, It has basic information on the users and listings like country,
town, address and if they are active or not. town, address and if they are active or not.
@ -1941,8 +1905,7 @@ models:
- name: check_in_cover_added_date_utc - name: check_in_cover_added_date_utc
data_type: date data_type: date
description: description: Date the user first included check-in cover
Date the user first included check-in cover
- name: billing_town - name: billing_town
data_type: character varying data_type: character varying
@ -1985,13 +1948,11 @@ models:
- name: check_in_cover_purchased - name: check_in_cover_purchased
data_type: bigint data_type: bigint
description: description: "Count of how many Check-in covers have been
"Count of how many Check-in covers have been
purchased for this accommodation" purchased for this accommodation"
- name: int_core__host_booking_fees - name: int_core__host_booking_fees
description: description: Bookings that have been processed by the Superhog backend.
Bookings that have been processed by the Superhog backend.
Each record matches one booking and has information on host Each record matches one booking and has information on host
booking fees, when they were charged and the currency used by booking fees, when they were charged and the currency used by
the host. the host.
@ -2031,18 +1992,18 @@ models:
- IncompleteInformation" - IncompleteInformation"
tests: tests:
- accepted_values: - accepted_values:
values: values:
- 'Approved' - "Approved"
- 'NotApproved' - "NotApproved"
- 'Cancelled' - "Cancelled"
- 'Rejected' - "Rejected"
- 'NoFlags' - "NoFlags"
- 'Flagged' - "Flagged"
- 'IncompleteInformation' - "IncompleteInformation"
- name: is_duplicate_booking - name: is_duplicate_booking
data_type: boolean data_type: boolean
description: | description: |
A flag that identifies whether the booking is a duplicate. A flag that identifies whether the booking is a duplicate.
A booking is considered a duplicate if there's an older booking with the same user, A booking is considered a duplicate if there's an older booking with the same user,
accomodation and check-in date. If there are two or more bookings with the same user, accomodation and check-in date. If there are two or more bookings with the same user,
@ -2114,7 +2075,7 @@ models:
description: The name of the role a user has. description: The name of the role a user has.
tests: tests:
- accepted_values: - accepted_values:
values: values:
- Host - Host
- Platform - Platform
- EDeposit - EDeposit
@ -2125,24 +2086,24 @@ models:
- PropertyVerificationManager - PropertyVerificationManager
- name: int_core__user_host - name: int_core__user_host
description: | description: |
This table provides information of the users that act as Hosts. This table provides information of the users that act as Hosts.
A Host needs to be understood in the broad sense of the term. Here host means any A Host needs to be understood in the broad sense of the term. Here host means any
user that acts as a "B2B" client. user that acts as a "B2B" client.
The categorisation as a Host is based on two possibilities: the role of the user or if The categorisation as a Host is based on two possibilities: the role of the user or if
the user comes from Know Your Guest (KYG) within the claim table. Any user that has the user comes from Know Your Guest (KYG) within the claim table. Any user that has
any of the following roles will be considered as a Host in this table: any of the following roles will be considered as a Host in this table:
- Host - Host
- Platform - Platform
- EDeposit - EDeposit
- KnowYourGuest - KnowYourGuest
- ScreeningAPI - ScreeningAPI
- PropertyVerificationManager - PropertyVerificationManager
Additionally, any user that has any of these claim types will be considered as a Host: Additionally, any user that has any of these claim types will be considered as a Host:
- KygRegistrationSignUpType - KygRegistrationSignUpType
- KygRegistrationIntegrationTypeName - KygRegistrationIntegrationTypeName
- KygMvp - KygMvp
Lastly, in case a user satisfies multiple conditions, it will only appear once in this table. Lastly, in case a user satisfies multiple conditions, it will only appear once in this table.
columns: columns:
- name: id_user_host - name: id_user_host
@ -2260,12 +2221,12 @@ models:
this date. this date.
- name: int_core__user_migration - name: int_core__user_migration
description: | description: |
This table provides information of the host users that have been migrated. This table provides information of the host users that have been migrated.
At this stage, the main objective is to account for the user migration within At this stage, the main objective is to account for the user migration within
the scope of New Dashboard migration. the scope of New Dashboard migration.
It uses the migration configuration settled in the macro: It uses the migration configuration settled in the macro:
- user_migration_configuration -> get_new_dash_migration_phases_config - user_migration_configuration -> get_new_dash_migration_phases_config
columns: columns:
- name: id_user_host - name: id_user_host
@ -2723,7 +2684,6 @@ models:
description: | description: |
Timestamp of when this row was ingested from the Backend to the DWH. Timestamp of when this row was ingested from the Backend to the DWH.
- name: int_core__new_dash_user_overview - name: int_core__new_dash_user_overview
description: | description: |
This model allows for minimum tracking of how the new dash initative is This model allows for minimum tracking of how the new dash initative is
@ -2875,7 +2835,6 @@ models:
tests: tests:
- not_null - not_null
columns:
- name: id_user_host - name: id_user_host
data_type: string data_type: string
description: | description: |
@ -2884,7 +2843,6 @@ models:
tests: tests:
- not_null - not_null
columns:
- name: price_plan_charged_by_type - name: price_plan_charged_by_type
data_type: string data_type: string
description: | description: |
@ -2893,7 +2851,6 @@ models:
tests: tests:
- not_null - not_null
columns:
- name: price_plan_start_at_utc - name: price_plan_start_at_utc
data_type: timestamp data_type: timestamp
description: | description: |
@ -2902,7 +2859,6 @@ models:
tests: tests:
- not_null - not_null
columns:
- name: price_plan_end_at_utc - name: price_plan_end_at_utc
data_type: timestamp data_type: timestamp
description: | description: |
@ -2912,7 +2868,6 @@ models:
tests: tests:
- not_null - not_null
columns:
- name: price_plan_created_at_utc - name: price_plan_created_at_utc
data_type: timestamp data_type: timestamp
description: | description: |
@ -2921,7 +2876,6 @@ models:
tests: tests:
- not_null - not_null
columns:
- name: active_in_month_start_date_utc - name: active_in_month_start_date_utc
data_type: date data_type: date
description: | description: |
@ -2933,7 +2887,6 @@ models:
tests: tests:
- not_null - not_null
columns:
- name: active_in_month_end_date_utc - name: active_in_month_end_date_utc
data_type: date data_type: date
description: | description: |
@ -2945,7 +2898,6 @@ models:
tests: tests:
- not_null - not_null
columns:
- name: price_plans_active_in_month - name: price_plans_active_in_month
data_type: integer data_type: integer
description: | description: |
@ -2959,20 +2911,20 @@ models:
- not_null - not_null
- name: int_core__deal - name: int_core__deal
description: | description: |
This table provides information about the unique entity that identifies a This table provides information about the unique entity that identifies a
client, which is a Deal. client, which is a Deal.
A Deal is a common way to match information between Core, Xero and Hubspot A Deal is a common way to match information between Core, Xero and Hubspot
data sources. It's different from the typical User Host in the sense that data sources. It's different from the typical User Host in the sense that
a Deal can have multiple User accounts (usually referred to as Platform a Deal can have multiple User accounts (usually referred to as Platform
accounts). This is because in the past, different Host configurations could accounts). This is because in the past, different Host configurations could
only be done if multiple users were created. only be done if multiple users were created.
It can happen that a Deal has 1 or multiple hosts, as mentioned above. At It can happen that a Deal has 1 or multiple hosts, as mentioned above. At
the same time, not all users that act as hosts have a deal associated. One the same time, not all users that act as hosts have a deal associated. One
example is Know Your Guest (KYG) Lite accounts. However, there's also historical example is Know Your Guest (KYG) Lite accounts. However, there's also historical
cases that for whatever reason there's no Deal associated. cases that for whatever reason there's no Deal associated.
For this model, the billing country and the deal name are estimated based on For this model, the billing country and the deal name are estimated based on
the information available in int_core__unified_user. the information available in int_core__unified_user.
columns: columns:
- name: id_deal - name: id_deal

11
models/intermediate/cross/schema.yml
View file

@ -164,7 +164,7 @@ models:
description: The dimension or granularity of the metrics. description: The dimension or granularity of the metrics.
tests: tests:
- accepted_values: - accepted_values:
values: values:
- global - global
- by_number_of_listings - by_number_of_listings
- by_billing_country - by_billing_country
@ -386,7 +386,7 @@ models:
description: The dimension or granularity of the metrics. description: The dimension or granularity of the metrics.
tests: tests:
- accepted_values: - accepted_values:
values: values:
- global - global
- by_number_of_listings - by_number_of_listings
- by_billing_country - by_billing_country
@ -420,8 +420,8 @@ models:
data_type: text data_type: text
description: allows for grouping and formatting for displaying purposes. description: allows for grouping and formatting for displaying purposes.
tests: tests:
- accepted_values: - accepted_values:
values: ['integer', 'percentage', 'currency_gbp'] values: ["integer", "percentage", "currency_gbp"]
- name: value - name: value
data_type: numeric data_type: numeric
@ -448,7 +448,6 @@ models:
positive impact for Superhog, otherwise is equal to relative_increment. positive impact for Superhog, otherwise is equal to relative_increment.
This value is specially created for formatting in PBI This value is specially created for formatting in PBI
- name: int_monthly_aggregated_metrics_history_by_deal - name: int_monthly_aggregated_metrics_history_by_deal
description: | description: |
This model aggregates the monthly historic information regarding the different metrics computed This model aggregates the monthly historic information regarding the different metrics computed
@ -574,7 +573,7 @@ models:
description: The dimension or granularity of the metrics. description: The dimension or granularity of the metrics.
tests: tests:
- accepted_values: - accepted_values:
values: values:
- global - global
- by_number_of_listings - by_number_of_listings
- by_billing_country - by_billing_country

37
models/intermediate/edeposit/schema.yaml → models/intermediate/edeposit/schema.yml
View file

@ -21,8 +21,7 @@ models:
- name: id_booking - name: id_booking
data_type: text data_type: text
description: description: "unique Superhog generated id for a booking.
"unique Superhog generated id for a booking.
note that this could be duplicated and both will be charged, note that this could be duplicated and both will be charged,
it's up to the user to no generate duplicate verifications" it's up to the user to no generate duplicate verifications"
@ -38,21 +37,19 @@ models:
- name: version - name: version
data_type: text data_type: text
description: description: "value to identify if it is Guesty (V1) or E-deposit (V2)"
"value to identify if it is Guesty (V1) or E-deposit (V2)"
tests: tests:
- accepted_values: - accepted_values:
values: values:
- V1 - V1
- V2 - V2
- name: verification_source - name: verification_source
data_type: text data_type: text
description: description: "source of the verification for the booking"
"source of the verification for the booking"
tests: tests:
- accepted_values: - accepted_values:
values: values:
- Guesty - Guesty
- Edeposit - Edeposit
@ -197,8 +194,7 @@ models:
- name: athena_creation_date_utc - name: athena_creation_date_utc
data_type: date data_type: date
description: description: "Athena date referring to when the booking was created.
"Athena date referring to when the booking was created.
It's provided by Guesty, but is not mandatory. It's provided by Guesty, but is not mandatory.
In case of doubt use created_at_utc or created_date_utc fields" In case of doubt use created_at_utc or created_date_utc fields"
@ -220,8 +216,7 @@ models:
columns: columns:
- name: id_verification - name: id_verification
data_type: text data_type: text
description: description: "Unique Superhog generated id for this verification.
"Unique Superhog generated id for this verification.
Note that there are some users that have a different id in Cosmos. Note that there are some users that have a different id in Cosmos.
For those users we created a mapping to relate this ids." For those users we created a mapping to relate this ids."
tests: tests:
@ -230,8 +225,7 @@ models:
- name: id_booking - name: id_booking
data_type: text data_type: text
description: description: "unique Superhog generated id for a booking.
"unique Superhog generated id for a booking.
note that this could be duplicated and both will be charged, note that this could be duplicated and both will be charged,
it's up to the user to no generate duplicate verifications" it's up to the user to no generate duplicate verifications"
tests: tests:
@ -310,8 +304,7 @@ models:
- not_null - not_null
- name: int_edeposit__guesty_verifications - name: int_edeposit__guesty_verifications
description: description: "This table shows all verification for Guesty.
"This table shows all verification for Guesty.
The charged fee is 2GBP per booked night if booking is approved The charged fee is 2GBP per booked night if booking is approved
(considered 1 night when the checkin and checkout are on the same day), (considered 1 night when the checkin and checkout are on the same day),
to be charged on checkout." to be charged on checkout."
@ -325,8 +318,7 @@ models:
- name: id_booking - name: id_booking
data_type: text data_type: text
description: description: "unique Superhog generated id for a booking.
"unique Superhog generated id for a booking.
note that this could be duplicated and both will be charged, note that this could be duplicated and both will be charged,
it's up to the user to not generate or cancel duplicate verifications" it's up to the user to not generate or cancel duplicate verifications"
tests: tests:
@ -344,8 +336,7 @@ models:
- name: ok_status_fee_in_gbp - name: ok_status_fee_in_gbp
data_type: integer data_type: integer
description: description: "total fee charged on checkout, this is only charged for approved verifications"
"total fee charged on checkout, this is only charged for approved verifications"
tests: tests:
- not_null - not_null
- dbt_expectations.expect_column_values_to_be_between: - dbt_expectations.expect_column_values_to_be_between:
@ -355,14 +346,12 @@ models:
- name: created_date_utc - name: created_date_utc
data_type: date data_type: date
description: description: "Date of creation of the verification in the system"
"Date of creation of the verification in the system"
tests: tests:
- not_null - not_null
- name: checkout_date_utc - name: checkout_date_utc
data_type: date data_type: date
description: description: "Date of checkout for the booking"
"Date of checkout for the booking"
tests: tests:
- not_null - not_null

0
models/intermediate/xero/schema.yaml → models/intermediate/xero/schema.yml
View file

110
models/reporting/core/schema.yaml → models/reporting/core/schema.yml
View file

@ -246,8 +246,7 @@ models:
- name: address_validation_passed - name: address_validation_passed
data_type: boolean data_type: boolean
description: description: "Boolean to indicate if the validation was approved or not.
"Boolean to indicate if the validation was approved or not.
In case there was a problem with the validation it will be Null" In case there was a problem with the validation it will be Null"
- name: address_validation_notes - name: address_validation_notes
@ -470,43 +469,35 @@ models:
- name: verification_start_at_utc - name: verification_start_at_utc
data_type: timestamp without time zone data_type: timestamp without time zone
description: description: The date and time at which the guest started the guest journey.
The date and time at which the guest started the guest journey.
- name: verification_start_date_utc - name: verification_start_date_utc
data_type: date data_type: date
description: description: The date on which the guest started the guest journey.
The date on which the guest started the guest journey.
- name: verification_end_at_utc - name: verification_end_at_utc
data_type: timestamp without time zone data_type: timestamp without time zone
description: description: The date and time at which the guest finished the guest journey.
The date and time at which the guest finished the guest journey.
- name: verification_end_date_utc - name: verification_end_date_utc
data_type: date data_type: date
description: description: The date on which the guest finished the guest journey.
The date on which the guest finished the guest journey.
- name: link_used_at_utc - name: link_used_at_utc
data_type: timestamp without time zone data_type: timestamp without time zone
description: description: The date and time at which the guest used the link for the verification.
The date and time at which the guest used the link for the verification.
- name: link_used_date_utc - name: link_used_date_utc
data_type: date data_type: date
description: description: The date on which the guest used the link for the verification.
The date on which the guest used the link for the verification.
- name: expire_at_utc - name: expire_at_utc
data_type: timestamp without time zone data_type: timestamp without time zone
description: description: The date and time at which the link for the verification expires.
The date and time at which the link for the verification expires.
- name: expire_date_utc - name: expire_date_utc
data_type: date data_type: date
description: description: The date on which the link for the verification expires.
The date on which the link for the verification expires.
- name: is_deleted - name: is_deleted
data_type: boolean data_type: boolean
@ -532,8 +523,7 @@ models:
- name: rejection_reason - name: rejection_reason
data_type: character varying data_type: character varying
description: description: Reason as to why the guest was rejected.
Reason as to why the guest was rejected.
- name: has_switched_to_mobile - name: has_switched_to_mobile
data_type: boolean data_type: boolean
@ -555,23 +545,19 @@ models:
- name: created_at_utc - name: created_at_utc
data_type: timestamp without time zone data_type: timestamp without time zone
description: description: The date and time at which the verification process was created.
The date and time at which the verification process was created.
- name: created_date_utc - name: created_date_utc
data_type: date data_type: date
description: description: The date on which the verification process was created.
The date on which the verification process was created.
- name: updated_at_utc - name: updated_at_utc
data_type: timestamp without time zone data_type: timestamp without time zone
description: description: The date and time at which the last update on the entry happened.
The date and time at which the last update on the entry happened.
- name: updated_date_utc - name: updated_date_utc
data_type: date data_type: date
description: description: The date on which the last update on the entry happened.
The date on which the last update on the entry happened.
- name: dwh_extracted_at_utc - name: dwh_extracted_at_utc
data_type: timestamp with time zone data_type: timestamp with time zone
@ -632,13 +618,11 @@ models:
- name: joined_at_utc - name: joined_at_utc
data_type: timestamp without time zone data_type: timestamp without time zone
description: description: The date time when the user created it's account in Superhog.
The date time when the user created it's account in Superhog.
- name: joined_date_utc - name: joined_date_utc
data_type: date data_type: date
description: description: The date when the user created it's account in Superhog.
The date when the user created it's account in Superhog.
- name: user_name - name: user_name
data_type: character varying data_type: character varying
@ -694,13 +678,11 @@ models:
- name: created_date_utc - name: created_date_utc
data_type: timestamp without time zone data_type: timestamp without time zone
description: description: The date time when the user data was created.
The date time when the user data was created.
- name: updated_date_utc - name: updated_date_utc
data_type: timestamp without time zone data_type: timestamp without time zone
description: description: The date when the user data was created.
The date when the user data was created.
- name: verified_at_utc - name: verified_at_utc
data_type: timestamp without time zone data_type: timestamp without time zone
@ -783,8 +765,7 @@ models:
- name: check_in_cover_added_date_utc - name: check_in_cover_added_date_utc
data_type: date data_type: date
description: description: Date the user first included check-in cover
Date the user first included check-in cover
- name: billing_town - name: billing_town
data_type: character varying data_type: character varying
@ -818,28 +799,25 @@ models:
- name: verification_request_booking_source - name: verification_request_booking_source
data_type: text data_type: text
description: description: Source type of host of the booking, this could be either;
Source type of host of the booking, this could be either;
- PMS - PMS
- OSL - OSL
- API/MANUAL - API/MANUAL
tests: tests:
- not_null - not_null
- accepted_values: - accepted_values:
values: values:
- 'PMS' - "PMS"
- 'OSL' - "OSL"
- 'API/MANUAL' - "API/MANUAL"
- name: experience_rating - name: experience_rating
data_type: bigint data_type: bigint
description: description: Guest rating of their experience with Superhog from 1 to 5
Guest rating of their experience with Superhog from 1 to 5
- name: guest_comments - name: guest_comments
data_type: character varying data_type: character varying
description: description: Guest comments on their experience with Superhog
Guest comments on their experience with Superhog
- name: is_contactable - name: is_contactable
data_type: boolean data_type: boolean
@ -848,13 +826,11 @@ models:
- name: created_at_utc - name: created_at_utc
data_type: timestamp without time zone data_type: timestamp without time zone
description: description: Date and time of response creation
Date and time of response creation
- name: updated_at_utc - name: updated_at_utc
data_type: timestamp without time zone data_type: timestamp without time zone
description: description: Date and time of last update of response
Date and time of last update of response
- name: selected_payment_option - name: selected_payment_option
data_type: character varying data_type: character varying
@ -889,8 +865,7 @@ models:
True if guest payed the fee True if guest payed the fee
- name: core__check_in_cover_listings - name: core__check_in_cover_listings
description: description: This model contains information about hosts and their listings
This model contains information about hosts and their listings
that offer check in cover. that offer check in cover.
It has basic information on the users and listings like country, It has basic information on the users and listings like country,
town, address and if they are active or not. town, address and if they are active or not.
@ -938,8 +913,7 @@ models:
- name: check_in_cover_added_date_utc - name: check_in_cover_added_date_utc
data_type: date data_type: date
description: description: Date the user first included check-in cover
Date the user first included check-in cover
- name: billing_town - name: billing_town
data_type: character varying data_type: character varying
@ -982,13 +956,11 @@ models:
- name: check_in_cover_purchased - name: check_in_cover_purchased
data_type: bigint data_type: bigint
description: description: "Count of how many Check-in covers have been
"Count of how many Check-in covers have been
purchased for this accommodation" purchased for this accommodation"
- name: core__host_booking_fees - name: core__host_booking_fees
description: description: Bookings that have been processed by the Superhog backend.
Bookings that have been processed by the Superhog backend.
Each record matches one booking and has information on host Each record matches one booking and has information on host
booking fees, when they were charged and the currency used by booking fees, when they were charged and the currency used by
the host. the host.
@ -1028,18 +1000,18 @@ models:
- IncompleteInformation" - IncompleteInformation"
tests: tests:
- accepted_values: - accepted_values:
values: values:
- 'Approved' - "Approved"
- 'NotApproved' - "NotApproved"
- 'Cancelled' - "Cancelled"
- 'Rejected' - "Rejected"
- 'NoFlags' - "NoFlags"
- 'Flagged' - "Flagged"
- 'IncompleteInformation' - "IncompleteInformation"
- name: is_duplicate_booking - name: is_duplicate_booking
data_type: boolean data_type: boolean
description: | description: |
A flag that identifies whether the booking is a duplicate. A flag that identifies whether the booking is a duplicate.
A booking is considered a duplicate if there's an older booking with the same user, A booking is considered a duplicate if there's an older booking with the same user,
accomodation and check-in date. If there are two or more bookings with the same user, accomodation and check-in date. If there are two or more bookings with the same user,

19
models/reporting/edeposit/schema.yaml → models/reporting/edeposit/schema.yml
View file

@ -1,7 +1,6 @@
version: 2 version: 2
models: models:
- name: edeposit__verification_fees - name: edeposit__verification_fees
description: description:
"This table shows all fee charges per verification for E-deposit. "This table shows all fee charges per verification for E-deposit.
@ -19,8 +18,7 @@ models:
- name: id_booking - name: id_booking
data_type: text data_type: text
description: description: "unique Superhog generated id for a booking.
"unique Superhog generated id for a booking.
note that this could be duplicated and both will be charged, note that this could be duplicated and both will be charged,
it's up to the user to no generate duplicate verifications" it's up to the user to no generate duplicate verifications"
tests: tests:
@ -99,8 +97,7 @@ models:
- not_null - not_null
- name: edeposit__guesty_verifications - name: edeposit__guesty_verifications
description: description: "This table shows all verification for Guesty.
"This table shows all verification for Guesty.
The charged fee is 2GBP per booked night if booking is approved The charged fee is 2GBP per booked night if booking is approved
(considered 1 night when the checkin and checkout are on the same day), (considered 1 night when the checkin and checkout are on the same day),
to be charged on checkout." to be charged on checkout."
@ -114,8 +111,7 @@ models:
- name: id_booking - name: id_booking
data_type: text data_type: text
description: description: "unique Superhog generated id for a booking.
"unique Superhog generated id for a booking.
note that this could be duplicated and both will be charged, note that this could be duplicated and both will be charged,
it's up to the user to not generate or cancel duplicate verifications" it's up to the user to not generate or cancel duplicate verifications"
tests: tests:
@ -133,8 +129,7 @@ models:
- name: ok_status_fee_in_gbp - name: ok_status_fee_in_gbp
data_type: integer data_type: integer
description: description: "total fee charged on checkout, this is only charged for approved verifications"
"total fee charged on checkout, this is only charged for approved verifications"
tests: tests:
- not_null - not_null
- dbt_expectations.expect_column_values_to_be_between: - dbt_expectations.expect_column_values_to_be_between:
@ -144,14 +139,12 @@ models:
- name: created_date_utc - name: created_date_utc
data_type: date data_type: date
description: description: "Date of creation of the verification in the system"
"Date of creation of the verification in the system"
tests: tests:
- not_null - not_null
- name: checkout_date_utc - name: checkout_date_utc
data_type: date data_type: date
description: description: "Date of checkout for the booking"
"Date of checkout for the booking"
tests: tests:
- not_null - not_null

3
models/reporting/exposures.yaml → models/reporting/exposures.yml
View file

@ -42,7 +42,8 @@ exposures:
label: E-deposit and Guesty Fees label: E-deposit and Guesty Fees
type: dashboard type: dashboard
maturity: low maturity: low
url: https://app.powerbi.com/groups/me/apps/33e55130-3a65-4fe8-86f2-11979fb2258a/reports/0642f366-c243-4879-8228-d8d6cc78f266/ReportSectionbd92a560d1aa856ba993?ctid=862842df-2998-4826-bea9-b726bc01d3a7&experience=power-bi url:
https://app.powerbi.com/groups/me/apps/33e55130-3a65-4fe8-86f2-11979fb2258a/reports/0642f366-c243-4879-8228-d8d6cc78f266/ReportSectionbd92a560d1aa856ba993?ctid=862842df-2998-4826-bea9-b726bc01d3a7&experience=power-bi
A PBI report showcasing our revenue coming from E-deposits and Guesty. A PBI report showcasing our revenue coming from E-deposits and Guesty.
Includes data from Xero for Guesty invoicing, and data from CosmosDB Includes data from Xero for Guesty invoicing, and data from CosmosDB

4
models/reporting/general/schema.yaml → models/reporting/general/schema.yml
View file

@ -404,8 +404,8 @@ models:
data_type: text data_type: text
description: Allows for grouping and formatting for displaying purposes. description: Allows for grouping and formatting for displaying purposes.
tests: tests:
- accepted_values: - accepted_values:
values: ['integer', 'percentage', 'currency_gbp'] values: ["integer", "percentage", "currency_gbp"]
- name: value - name: value
data_type: numeric data_type: numeric

6
models/reporting/xero/schema.yaml → models/reporting/xero/schema.yml
View file

@ -1072,7 +1072,7 @@ models:
- name: line_amount_wo_taxes_in_gbp - name: line_amount_wo_taxes_in_gbp
data_type: numeric data_type: numeric
description: | description: |
The total value transferred in this line, minus taxes, presented in The total value transferred in this line, minus taxes, presented in
GBP. GBP.
@ -1172,8 +1172,8 @@ models:
- not_null - not_null
- accepted_values: - accepted_values:
values: values:
- 'invoice' - "invoice"
- 'credit note' - "credit note"
- name: item_code - name: item_code
data_type: text data_type: text

6
models/staging/core/schema.yml
View file

@ -195,8 +195,7 @@ models:
- unique - unique
- not_null - not_null
- name: stg_core__edeposit_user - name: stg_core__edeposit_user
description: description: "This table contains data on partner users for E-deposit,
"This table contains data on partner users for E-deposit,
their currencies and amount of protection according to the their currencies and amount of protection according to the
level given on the verification" level given on the verification"
columns: columns:
@ -216,8 +215,7 @@ models:
- name: currency - name: currency
data_type: character varying data_type: character varying
description: description: "Three-letter ISO code assigned to the currency used by user."
"Three-letter ISO code assigned to the currency used by user."
tests: tests:
- not_null - not_null
- dbt_expectations.expect_column_values_to_match_regex: - dbt_expectations.expect_column_values_to_match_regex:

6
models/staging/edeposit/schema.yml
View file

@ -4,7 +4,7 @@ models:
- name: stg_edeposit__verifications - name: stg_edeposit__verifications
description: description:
"Records of each transaction that happens in the edeposit API. Records are "Records of each transaction that happens in the edeposit API. Records are
mutable and can get updated." mutable and can get updated."
columns: columns:
- name: id_verification - name: id_verification
data_type: character varying data_type: character varying
@ -39,8 +39,8 @@ models:
- not_null - not_null
- accepted_values: - accepted_values:
values: values:
- 'V1' - "V1"
- 'V2' - "V2"
- name: nightly_fee_local - name: nightly_fee_local
data_type: numeric data_type: numeric

2
models/staging/xero/schema.yml
View file

@ -747,7 +747,7 @@ models:
- name: full_contact_details - name: full_contact_details
data_type: jsonb data_type: jsonb
description: | description: |
The contact related to this transaction. The contact related to this transaction.
Might be null since not all transactions have a related contact. Might be null since not all transactions have a related contact.

0
seeds/schema.yaml → seeds/schema.yml
View file