counterweight/data-dwh-dbt-project
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merged PR 2832: Format YAML on save + complements

Browse source 
# Description

This PR:
- Adds some `vscode` settings that make VSCode autoformat yaml files on save, like we do with `.sql` files. This should help with formatting consistency.
- I also made some notes about this on the readme.
- Finally, I made a couple of tiny unrelated changes on the readme since I was there.

Related work items: #21344
This commit is contained in:
Pablo Martín 2024-09-13 08:33:26 +00:00
commit ac1aed029d
15 changed files with 477 additions and 567 deletions
Download patch file Download diff file Expand all files Collapse all files

3
.vscode/settings.json vendored
View file

@ -12,6 +12,9 @@
},
"dbt.enableNewLineagePanel": true,
"dbt.enableNewDocsPanel": true,
"[yaml]": {
"editor.formatOnSave": true
},
"yaml.schemas": {
"https://raw.githubusercontent.com/dbt-labs/dbt-jsonschema/main/schemas/dbt_yml_files.json": [
"/**/*.yml",

11
README.md
View file

@ -51,9 +51,11 @@ pg_restore -h localhost -U postgres -W -d dwh xero.dump
This repo works in a trunk-based-development philosophy (<https://trunkbaseddevelopment.com/>).
When working on Data modeling stuff (models, sources, seeds, docs, etc.) use a `models` branch (i.e. `models/churned-users`). It's fine and encouraged to build incrementally towards a `reporting` level table with multiple PRs as long as you keep the model buildable along the way.
If your branch is related to a work item from DevOps, we encourage adding the ticket number in the branch name. For example: `models/123-some-fancy-name`. If you don't have a ticket number, you can simply do a `NOTICKET` one: `models/NOTICKET-some-fancy-name`.
For other matters, use a `chores` branch (i.e. `chores/add-dbt-package`).
When working on Data modeling stuff (models, sources, seeds, docs, etc.) use a `models` branch (i.e. `models/782-churned-users`). It's fine and encouraged to build incrementally towards a `reporting` level table with multiple PRs as long as you keep the model buildable along the way.
For other matters, use a `chores` branch (i.e. `chores/656-add-dbt-package`).
## Project organization
@ -89,6 +91,9 @@ We organize models in three folders:
- Split schema and domain with double underscode (ie `stg_core__booking`).
- Always use sources to read into staging models.
- SQL formatting should be done with `sqlfmt`.
- YAML files:
- Should use the `.yml` extension, not `.yaml`.
- Should be autoformatted on save. If you install [this vscode extension](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml), autoformatting should happen out of the box thanks to the settings included in the `.vscode/settings.json` file.
- Other conventions
- In staging, enforce a `lower()` to user UUID fields to avoid nasty propagations in the DWH.
@ -131,6 +136,4 @@ To maintain:
## Stuff that we haven't done but we would like to
- Automate formatting with git pre-commit.
- Define conventions on testing (and enforce them).
- Define conventions on documentation (and enforce them).
- Prepare a quick way to replicate parts of the `prd` dwh in our local machines.

572
models/intermediate/core/schema.yaml → models/intermediate/core/schema.yml
View file

File diff suppressed because it is too large Load diff

69
models/intermediate/cross/schema.yml
View file

@ -140,18 +140,18 @@ models:
and computes any necessary weighted metric across different sources.
Each metric has a date, dimension and dimension value that defines
the primary key of this model.
Finally, it displays any metric on the current date, the previous year
date and it computes the relative increment by using the macro:
- calculate_safe_relative_increment
tests:
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- date
- dimension
- dimension_value
columns:
- name: date
data_type: date
@ -164,11 +164,11 @@ models:
description: The dimension or granularity of the metrics.
tests:
- accepted_values:
values:
values:
- global
- by_number_of_listings
- by_billing_country
- name: dimension_value
data_type: string
description: The value or segment available for the selected dimension.
@ -197,7 +197,7 @@ models:
- not_null
- name: month
data_type: int
data_type: int
description: Month number of the given date.
tests:
- not_null
@ -249,7 +249,7 @@ models:
combination_of_columns:
- date
- id_deal
columns:
- name: year
data_type: int
@ -258,7 +258,7 @@ models:
- not_null
- name: month
data_type: int
data_type: int
description: Month number of the given date.
tests:
- not_null
@ -324,14 +324,14 @@ models:
a set of metric, value, previous_year_value and relative_increment at a given date. It uses Jinja
code to avoid code replication.
tests:
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- date
- metric
- dimension
- dimension_value
columns:
- name: year
data_type: int
@ -340,7 +340,7 @@ models:
- not_null
- name: month
data_type: int
data_type: int
description: month number of the given date.
tests:
- not_null
@ -386,11 +386,11 @@ models:
description: The dimension or granularity of the metrics.
tests:
- accepted_values:
values:
values:
- global
- by_number_of_listings
- by_billing_country
- name: dimension_value
data_type: string
description: The value or segment available for the selected dimension.
@ -420,35 +420,34 @@ models:
data_type: text
description: allows for grouping and formatting for displaying purposes.
tests:
- accepted_values:
values: ['integer', 'percentage', 'currency_gbp']
- accepted_values:
values: ["integer", "percentage", "currency_gbp"]
- name: value
data_type: numeric
description: |
data_type: numeric
description: |
numeric value (integer or decimal) that corresponds to the MTD computation of the metric
at a given date.
- name: previous_year_value
data_type: numeric
description: |
data_type: numeric
description: |
numeric value (integer or decimal) that corresponds to the MTD computation of the metric
on the previous year at a given date.
- name: relative_increment
data_type: numeric
description: |
data_type: numeric
description: |
numeric value that corresponds to the relative increment between value and previous year value,
following the computation: value / previous_year_value - 1.
- name: relative_increment_with_sign_format
data_type: numeric
description: |
data_type: numeric
description: |
relative_increment value multiplied by -1 in case this metric's growth doesn't have a
positive impact for Superhog, otherwise is equal to relative_increment.
This value is specially created for formatting in PBI
- name: int_monthly_aggregated_metrics_history_by_deal
description: |
This model aggregates the monthly historic information regarding the different metrics computed
@ -463,9 +462,9 @@ models:
the int_mtd_aggregated metrics because 1) the mtd version contains more computing dates
than the by deal version, the latest being a subset of the first, and 2) the deal based model
enforces that a booking/guest journey/listing/etc has a host with a deal assigned, which is
not necessarily the case.
not necessarily the case.
tests:
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- date
@ -480,7 +479,7 @@ models:
- name: id_deal
data_type: character varying
description: Id of the deal associated to the host.
description: Id of the deal associated to the host.
tests:
- not_null
@ -496,7 +495,7 @@ models:
description: |
ISO 3166-1 alpha-3 main country code in which the Deal is billed.
In some cases it's null.
- name: int_dates_mtd_by_dimension
description: |
This model provides Month-To-Date (MTD) necessary dates, dimension and dimension_values
@ -504,7 +503,7 @@ models:
It provides the basic "empty" structure from which metrics will be built upon. This is, on
top of the Date that characterises int_dates_mtd, including the dimensions and their
respective values that should appear in any mtd metric model.
Example:
- For the "global" dimension, we will only have the "global" dimension value.
- For the "by_number_of_listing" dimension, we will have different values
@ -513,7 +512,7 @@ models:
... and so on and forth for any available dimension. These combinations should appear
for each date of the MTD models.
tests:
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- date
@ -528,7 +527,7 @@ models:
- not_null
- name: month
data_type: int
data_type: int
description: Month number of the given date.
tests:
- not_null
@ -565,7 +564,7 @@ models:
data_type: date
description: |
Main date for the computation, metrics include monthly information
until this date.
until this date.
tests:
- not_null
@ -574,13 +573,13 @@ models:
description: The dimension or granularity of the metrics.
tests:
- accepted_values:
values:
values:
- global
- by_number_of_listings
- by_billing_country
- name: dimension_value
data_type: string
description: The value or segment available for the selected dimension.
tests:
- not_null
- not_null

73
models/intermediate/edeposit/schema.yaml → models/intermediate/edeposit/schema.yml
View file

@ -2,28 +2,27 @@ version: 2
models:
- name: int_edeposit__verifications
description:
description:
"This table holds records on verifications for e-deposit bookings.
It contains details on validations checked on the guests, guest information
and some booking details like checkin-checkout date or the status of the verification.
The id values found here are completely unrelated to the ones found in Core DWH.
Note that id_verifications and booking_id should normally be 1 to 1.
Though there are exception, the API will accept a duplicate booking and the users
will be charged for it. A duplicate would return a unique id_verification."
columns:
- name: id_verification
data_type: text
description: "unique Superhog generated id for this verification"
description: "unique Superhog generated id for this verification"
tests:
- unique
- not_null
- name: id_booking
data_type: text
description:
"unique Superhog generated id for a booking.
note that this could be duplicated and both will be charged,
description: "unique Superhog generated id for a booking.
note that this could be duplicated and both will be charged,
it's up to the user to no generate duplicate verifications"
- name: id_user_partner
@ -38,21 +37,19 @@ models:
- name: version
data_type: text
description:
"value to identify if it is Guesty (V1) or E-deposit (V2)"
description: "value to identify if it is Guesty (V1) or E-deposit (V2)"
tests:
- accepted_values:
values:
values:
- V1
- V2
- name: verification_source
data_type: text
description:
"source of the verification for the booking"
description: "source of the verification for the booking"
tests:
- accepted_values:
values:
values:
- Guesty
- Edeposit
@ -190,16 +187,15 @@ models:
- name: athena_creation_at_utc
data_type: timestamp without time zone
description:
"Athena timestamp referring to when the booking was created.
It's provided by Guesty, but is not mandatory.
description:
"Athena timestamp referring to when the booking was created.
It's provided by Guesty, but is not mandatory.
In case of doubt use created_at_utc or created_date_utc fields"
- name: athena_creation_date_utc
data_type: date
description:
"Athena date referring to when the booking was created.
It's provided by Guesty, but is not mandatory.
description: "Athena date referring to when the booking was created.
It's provided by Guesty, but is not mandatory.
In case of doubt use created_at_utc or created_date_utc fields"
- name: created_at_utc
@ -211,7 +207,7 @@ models:
description: "Date of creation of the verification in the system"
- name: int_edeposit__verification_fees
description:
description:
"This table shows all fee charges per verification for E-deposit.
Cancellation fee is charged when the monthly rate of cancelled bookings over
total booking of the partner surpasses the threshold (currently set at 0.05).
@ -220,8 +216,7 @@ models:
columns:
- name: id_verification
data_type: text
description:
"Unique Superhog generated id for this verification.
description: "Unique Superhog generated id for this verification.
Note that there are some users that have a different id in Cosmos.
For those users we created a mapping to relate this ids."
tests:
@ -230,9 +225,8 @@ models:
- name: id_booking
data_type: text
description:
"unique Superhog generated id for a booking.
note that this could be duplicated and both will be charged,
description: "unique Superhog generated id for a booking.
note that this could be duplicated and both will be charged,
it's up to the user to no generate duplicate verifications"
tests:
- not_null
@ -281,7 +275,7 @@ models:
- name: cancelled_fee_in_txn_currency
data_type: numeric
description: "fee charged in used currency for cancelled verifications"
description: "fee charged in used currency for cancelled verifications"
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
@ -290,7 +284,7 @@ models:
- name: cancelled_fee_in_gbp
data_type: numeric
description: "fee charged in gbp for cancelled verifications"
description: "fee charged in gbp for cancelled verifications"
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
@ -299,21 +293,20 @@ models:
- name: checkout_date_utc
data_type: date
description: "Date of checkout for the booking"
description: "Date of checkout for the booking"
tests:
- not_null
- name: created_date_utc
data_type: date
description: "Date of creation of the verification in the system"
description: "Date of creation of the verification in the system"
tests:
- not_null
- name: int_edeposit__guesty_verifications
description:
"This table shows all verification for Guesty.
description: "This table shows all verification for Guesty.
The charged fee is 2GBP per booked night if booking is approved
(considered 1 night when the checkin and checkout are on the same day),
(considered 1 night when the checkin and checkout are on the same day),
to be charged on checkout."
columns:
- name: id_verification
@ -325,9 +318,8 @@ models:
- name: id_booking
data_type: text
description:
"unique Superhog generated id for a booking.
note that this could be duplicated and both will be charged,
description: "unique Superhog generated id for a booking.
note that this could be duplicated and both will be charged,
it's up to the user to not generate or cancel duplicate verifications"
tests:
- not_null
@ -344,8 +336,7 @@ models:
- name: ok_status_fee_in_gbp
data_type: integer
description:
"total fee charged on checkout, this is only charged for approved verifications"
description: "total fee charged on checkout, this is only charged for approved verifications"
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
@ -355,14 +346,12 @@ models:
- name: created_date_utc
data_type: date
description:
"Date of creation of the verification in the system"
description: "Date of creation of the verification in the system"
tests:
- not_null
- name: checkout_date_utc
data_type: date
description:
"Date of checkout for the booking"
description: "Date of checkout for the booking"
tests:
- not_null
- not_null

0
models/intermediate/xero/schema.yaml → models/intermediate/xero/schema.yml
View file

158
models/reporting/core/schema.yaml → models/reporting/core/schema.yml
View file

@ -161,7 +161,7 @@ models:
currency.
This fee is calculated by finding what was the active price plan for the host user
when the booking fee had to be charged.
when the booking fee had to be charged.
- name: booking_fee_charge_at_utc
data_type: timestamp without time zone
@ -191,7 +191,7 @@ models:
columns:
- name: id_verification_request
data_type: bigint
description: |
description: |
Unique, incremental, internal ID for the related verification
request.
tests:
@ -220,25 +220,25 @@ models:
- name: is_verification_request_complete
data_type: boolean
description: |
description: |
True if the verification request is considered is considered
complete, AKA the guest has finished the full guest journey.
- name: is_past_check_in
data_type: boolean
description: |
description: |
True if the check-in date is in the past, false otherwise. This value
only gets informed for users who have bought check-in cover.
- name: is_awaiting_check_in
data_type: boolean
description: |
description: |
True if the check-in date is in the future, false otherwise. This
value only gets informed for users who have bought check-in cover.
- name: cover_was_purchased
data_type: boolean
description: |
description: |
True if the guest purchased the check-in cover.
This is identified by the fact that there is a payment of type
@ -246,8 +246,7 @@ models:
- name: address_validation_passed
data_type: boolean
description:
"Boolean to indicate if the validation was approved or not.
description: "Boolean to indicate if the validation was approved or not.
In case there was a problem with the validation it will be Null"
- name: address_validation_notes
@ -307,7 +306,7 @@ models:
- name: verification_start_at_utc
data_type: timestamp without time zone
description: |
description: |
The date and time at which the guest started the guest journey.
- name: verification_start_date_utc
@ -326,7 +325,7 @@ models:
- name: amount_in_txn_currency
data_type: numeric
description: |
description: |
The amount paid by the guest for the check-in cover, in the currency
in which the payment actually took place.
@ -348,7 +347,7 @@ models:
- name: checkin_cover_cover_amount_local_curr
data_type: numeric
description: |
description: |
The total amount for which the guest check-in is covered, according
to the Superhog backend, in the same currency as the guest payment.
@ -385,18 +384,18 @@ models:
For more details, contact pablo.martin or Lawrence.
- name: core__verification_requests
description:
description:
This is a table that shows all guest journey from our guests users with
each record matching each guest journey.
It holds information about the guests like name, email, phone, etc.., as
well as dates regarding the process of the guest journey like when it
well as dates regarding the process of the guest journey like when it
was started or finished.
columns:
- name: id_verification_request
data_type: bigint
description:
Unique, incremental, internal ID for the related verification
description:
Unique, incremental, internal ID for the related verification
request.
- name: uuid_verification_request
@ -425,7 +424,7 @@ models:
- name: is_verification_request_complete
data_type: boolean
description: True if the verification request is considered
description: True if the verification request is considered
complete, AKA the guest has finished the full guest journey.
- name: verification_url
@ -470,43 +469,35 @@ models:
- name: verification_start_at_utc
data_type: timestamp without time zone
description:
The date and time at which the guest started the guest journey.
description: The date and time at which the guest started the guest journey.
- name: verification_start_date_utc
data_type: date
description:
The date on which the guest started the guest journey.
description: The date on which the guest started the guest journey.
- name: verification_end_at_utc
data_type: timestamp without time zone
description:
The date and time at which the guest finished the guest journey.
description: The date and time at which the guest finished the guest journey.
- name: verification_end_date_utc
data_type: date
description:
The date on which the guest finished the guest journey.
description: The date on which the guest finished the guest journey.
- name: link_used_at_utc
data_type: timestamp without time zone
description:
The date and time at which the guest used the link for the verification.
description: The date and time at which the guest used the link for the verification.
- name: link_used_date_utc
data_type: date
description:
The date on which the guest used the link for the verification.
description: The date on which the guest used the link for the verification.
- name: expire_at_utc
data_type: timestamp without time zone
description:
The date and time at which the link for the verification expires.
description: The date and time at which the link for the verification expires.
- name: expire_date_utc
data_type: date
description:
The date on which the link for the verification expires.
description: The date on which the link for the verification expires.
- name: is_deleted
data_type: boolean
@ -532,8 +523,7 @@ models:
- name: rejection_reason
data_type: character varying
description:
Reason as to why the guest was rejected.
description: Reason as to why the guest was rejected.
- name: has_switched_to_mobile
data_type: boolean
@ -555,30 +545,26 @@ models:
- name: created_at_utc
data_type: timestamp without time zone
description:
The date and time at which the verification process was created.
description: The date and time at which the verification process was created.
- name: created_date_utc
data_type: date
description:
The date on which the verification process was created.
description: The date on which the verification process was created.
- name: updated_at_utc
data_type: timestamp without time zone
description:
The date and time at which the last update on the entry happened.
description: The date and time at which the last update on the entry happened.
- name: updated_date_utc
data_type: date
description:
The date on which the last update on the entry happened.
description: The date on which the last update on the entry happened.
- name: dwh_extracted_at_utc
data_type: timestamp with time zone
description: ""
- name: core__unified_user
description:
description:
This is a table to join data from all users of Superhog (hosts, PMs, guests, etc).
It contains details of every user like name, last name, email address, etc.
@ -589,7 +575,7 @@ models:
tests:
- unique
- not_null
- name: id_account_type
data_type: bigint
description: Internal ID for type of user.
@ -632,13 +618,11 @@ models:
- name: joined_at_utc
data_type: timestamp without time zone
description:
The date time when the user created it's account in Superhog.
description: The date time when the user created it's account in Superhog.
- name: joined_date_utc
data_type: date
description:
The date when the user created it's account in Superhog.
description: The date when the user created it's account in Superhog.
- name: user_name
data_type: character varying
@ -694,13 +678,11 @@ models:
- name: created_date_utc
data_type: timestamp without time zone
description:
The date time when the user data was created.
description: The date time when the user data was created.
- name: updated_date_utc
data_type: timestamp without time zone
description:
The date when the user data was created.
description: The date when the user data was created.
- name: verified_at_utc
data_type: timestamp without time zone
@ -735,7 +717,7 @@ models:
description: ""
- name: core__check_in_cover_users
description:
description:
This model contains information about hosts that offer check in cover.
It has basic information on the users like name, phone, email or joined date.
@ -783,8 +765,7 @@ models:
- name: check_in_cover_added_date_utc
data_type: date
description:
Date the user first included check-in cover
description: Date the user first included check-in cover
- name: billing_town
data_type: character varying
@ -795,7 +776,7 @@ models:
description: ""
- name: core__guest_satisfaction_responses
description:
description:
This model contains information on guests satisfaction survey responses,
it contains some basic information on the guests, a rating of their experience
and some comments on it.
@ -818,28 +799,25 @@ models:
- name: verification_request_booking_source
data_type: text
description:
Source type of host of the booking, this could be either;
description: Source type of host of the booking, this could be either;
- PMS
- OSL
- API/MANUAL
tests:
- not_null
- accepted_values:
values:
- 'PMS'
- 'OSL'
- 'API/MANUAL'
values:
- "PMS"
- "OSL"
- "API/MANUAL"
- name: experience_rating
data_type: bigint
description:
Guest rating of their experience with Superhog from 1 to 5
description: Guest rating of their experience with Superhog from 1 to 5
- name: guest_comments
data_type: character varying
description:
Guest comments on their experience with Superhog
description: Guest comments on their experience with Superhog
- name: is_contactable
data_type: boolean
@ -848,13 +826,11 @@ models:
- name: created_at_utc
data_type: timestamp without time zone
description:
Date and time of response creation
description: Date and time of response creation
- name: updated_at_utc
data_type: timestamp without time zone
description:
Date and time of last update of response
description: Date and time of last update of response
- name: selected_payment_option
data_type: character varying
@ -889,13 +865,12 @@ models:
True if guest payed the fee
- name: core__check_in_cover_listings
description:
This model contains information about hosts and their listings
description: This model contains information about hosts and their listings
that offer check in cover.
It has basic information on the users and listings like country,
town, address and if they are active or not.
This model is restricted to active user so it doesn't include historical
This model is restricted to active user so it doesn't include historical
data like users that had check-in cover but are currently inactive.
columns:
- name: id_user_host
@ -938,8 +913,7 @@ models:
- name: check_in_cover_added_date_utc
data_type: date
description:
Date the user first included check-in cover
description: Date the user first included check-in cover
- name: billing_town
data_type: character varying
@ -982,15 +956,13 @@ models:
- name: check_in_cover_purchased
data_type: bigint
description:
"Count of how many Check-in covers have been
description: "Count of how many Check-in covers have been
purchased for this accommodation"
- name: core__host_booking_fees
description:
Bookings that have been processed by the Superhog backend.
description: Bookings that have been processed by the Superhog backend.
Each record matches one booking and has information on host
booking fees, when they were charged and the currency used by
booking fees, when they were charged and the currency used by
the host.
columns:
@ -1017,7 +989,7 @@ models:
- name: booking_state
data_type: character varying
description:
description:
"State in which the booking is, could be either of the following:
- Approved
- NotApproved
@ -1026,20 +998,20 @@ models:
- NoFlags
- Flagged
- IncompleteInformation"
tests:
tests:
- accepted_values:
values:
- 'Approved'
- 'NotApproved'
- 'Cancelled'
- 'Rejected'
- 'NoFlags'
- 'Flagged'
- 'IncompleteInformation'
values:
- "Approved"
- "NotApproved"
- "Cancelled"
- "Rejected"
- "NoFlags"
- "Flagged"
- "IncompleteInformation"
- name: is_duplicate_booking
data_type: boolean
description: |
description: |
A flag that identifies whether the booking is a duplicate.
A booking is considered a duplicate if there's an older booking with the same user,
accomodation and check-in date. If there are two or more bookings with the same user,
@ -1151,7 +1123,7 @@ models:
Count of listings that the user owns.
It contains only those that can accept bookings (hard
activation - not to be confused with activity-based
segmentation).
segmentation).
- name: total_listings_with_product_bundle_with_paid_service
data_type: integer
@ -1195,7 +1167,7 @@ models:
data_type: integer
description: |
Integer-based flag version of total_bookings_with_product_bundle.
- name: has_bookings_with_product_bundle_with_paid_service
data_type: integer
description: |

39
models/reporting/edeposit/schema.yaml → models/reporting/edeposit/schema.yml
View file

@ -1,9 +1,8 @@
version: 2
models:
- name: edeposit__verification_fees
description:
description:
"This table shows all fee charges per verification for E-deposit.
Cancellation fee is charged when the monthly rate of cancelled bookings over
total booking of the partner surpasses the threshold (currently set at 5%).
@ -19,9 +18,8 @@ models:
- name: id_booking
data_type: text
description:
"unique Superhog generated id for a booking.
note that this could be duplicated and both will be charged,
description: "unique Superhog generated id for a booking.
note that this could be duplicated and both will be charged,
it's up to the user to no generate duplicate verifications"
tests:
- not_null
@ -70,7 +68,7 @@ models:
- name: cancelled_fee_in_txn_currency
data_type: numeric
description: "fee charged in used currency for cancelled verifications"
description: "fee charged in used currency for cancelled verifications"
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
@ -79,7 +77,7 @@ models:
- name: cancelled_fee_in_gbp
data_type: numeric
description: "fee charged in gbp for cancelled verifications"
description: "fee charged in gbp for cancelled verifications"
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
@ -88,21 +86,20 @@ models:
- name: checkout_date_utc
data_type: date
description: "Date of checkout for the booking"
description: "Date of checkout for the booking"
tests:
- not_null
- name: created_date_utc
data_type: date
description: "Date of creation of the verification in the system"
description: "Date of creation of the verification in the system"
tests:
- not_null
- name: edeposit__guesty_verifications
description:
"This table shows all verification for Guesty.
description: "This table shows all verification for Guesty.
The charged fee is 2GBP per booked night if booking is approved
(considered 1 night when the checkin and checkout are on the same day),
(considered 1 night when the checkin and checkout are on the same day),
to be charged on checkout."
columns:
- name: id_verification
@ -114,9 +111,8 @@ models:
- name: id_booking
data_type: text
description:
"unique Superhog generated id for a booking.
note that this could be duplicated and both will be charged,
description: "unique Superhog generated id for a booking.
note that this could be duplicated and both will be charged,
it's up to the user to not generate or cancel duplicate verifications"
tests:
- not_null
@ -133,8 +129,7 @@ models:
- name: ok_status_fee_in_gbp
data_type: integer
description:
"total fee charged on checkout, this is only charged for approved verifications"
description: "total fee charged on checkout, this is only charged for approved verifications"
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
@ -144,14 +139,12 @@ models:
- name: created_date_utc
data_type: date
description:
"Date of creation of the verification in the system"
description: "Date of creation of the verification in the system"
tests:
- not_null
- name: checkout_date_utc
data_type: date
description:
"Date of checkout for the booking"
description: "Date of checkout for the booking"
tests:
- not_null
- not_null

11
models/reporting/exposures.yaml → models/reporting/exposures.yml
View file

@ -17,7 +17,7 @@ exposures:
owner:
name: Pablo Martin
email: pablo.martin@superhog.com
- name: host_fees
label: Host Fees
type: dashboard
@ -42,7 +42,8 @@ exposures:
label: E-deposit and Guesty Fees
type: dashboard
maturity: low
url: https://app.powerbi.com/groups/me/apps/33e55130-3a65-4fe8-86f2-11979fb2258a/reports/0642f366-c243-4879-8228-d8d6cc78f266/ReportSectionbd92a560d1aa856ba993?ctid=862842df-2998-4826-bea9-b726bc01d3a7&experience=power-bi
url:
https://app.powerbi.com/groups/me/apps/33e55130-3a65-4fe8-86f2-11979fb2258a/reports/0642f366-c243-4879-8228-d8d6cc78f266/ReportSectionbd92a560d1aa856ba993?ctid=862842df-2998-4826-bea9-b726bc01d3a7&experience=power-bi
A PBI report showcasing our revenue coming from E-deposits and Guesty.
Includes data from Xero for Guesty invoicing, and data from CosmosDB
@ -65,7 +66,7 @@ exposures:
description: |
A PBI report to monitor the performance of the Check-In Hero product.
Includes contents like Revenue generated, Outstanding Risk being
generated by the product and conversion funnels.
@ -104,7 +105,7 @@ exposures:
url: https://app.powerbi.com/links/ynCnuqVPTr?ctid=862842df-2998-4826-bea9-b726bc01d3a7&pbi_source=linkShare&bookmarkGuid=81e69a3e-90dc-46d7-b313-ba55ca611945
description: |
A PBI report showcasing the main business KPIs of our company.
Includes multiple high-level metrics updated in a daily basis,
ensuring both a Month-to-date comprehension of our business and
a view on historic values on a month-by-month approach.
@ -224,4 +225,4 @@ exposures:
owner:
name: Oriol Roqué
email: oriol.roque@superhog.com
email: oriol.roque@superhog.com

42
models/reporting/general/schema.yaml → models/reporting/general/schema.yml
View file

@ -166,7 +166,7 @@ models:
description: The full date for the last day of the year.
- name: daily_currency_exchange_rates
description:
description:
This model holds a lot of data on currency exchange rates. The time
granularity is daily. Each record holds a currency pair for a specific
day, source and version.
@ -306,7 +306,7 @@ models:
It's the main source of information for the Main KPIs reporting, specifically
on the MTD (Month To Date) and the Monthly Overview.
tests:
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- date
@ -322,7 +322,7 @@ models:
- not_null
- name: month
data_type: int
data_type: int
description: Month number of the given date.
tests:
- not_null
@ -370,11 +370,11 @@ models:
The dimension or granularity of the metrics. Keep in mind that
in this reporting model this field corresponds to the
dimension_display; this is, the name of the dimension for
displaying purposes.
displaying purposes.
tests:
- not_null
- name: dimension_value
data_type: string
description: The value or segment available for the selected dimension.
@ -404,31 +404,31 @@ models:
data_type: text
description: Allows for grouping and formatting for displaying purposes.
tests:
- accepted_values:
values: ['integer', 'percentage', 'currency_gbp']
- accepted_values:
values: ["integer", "percentage", "currency_gbp"]
- name: value
data_type: numeric
description: |
data_type: numeric
description: |
Numeric value (integer or decimal) that corresponds to the MTD computation of the metric
at a given date. Note that if the month is not in progress, then this value corresponds
to the monthly figure.
- name: previous_year_value
data_type: numeric
description: |
data_type: numeric
description: |
Numeric value (integer or decimal) that corresponds to the MTD computation of the metric
on the previous year at a given date.
- name: relative_increment
data_type: numeric
description: |
data_type: numeric
description: |
Numeric value that corresponds to the relative increment between value and previous year value,
following the computation: value / previous_year_value - 1.
- name: relative_increment_with_sign_format
data_type: numeric
description: |
data_type: numeric
description: |
Relative_increment value multiplied by -1 in case this metric's growth doesn't have a
positive impact for Superhog, otherwise is equal to relative_increment.
This value is specially created for formatting in PBI
@ -448,9 +448,9 @@ models:
the int_mtd_aggregated metrics because 1) the mtd version contains more computing dates
than the by deal version, the latest being a subset of the first, and 2) the deal based model
enforces that a booking/guest journey/listing/etc has a host with a deal assigned, which is
not necessarily the case.
not necessarily the case.
tests:
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- date
@ -463,10 +463,10 @@ models:
tests:
- not_null
- latest_date_is_yesterday
- name: id_deal
data_type: character varying
description: Id of the deal associated to the host.
description: Id of the deal associated to the host.
tests:
- not_null
@ -490,7 +490,7 @@ models:
- not_null
- name: month
data_type: int
data_type: int
description: month number of the given date.
tests:
- not_null
@ -499,4 +499,4 @@ models:
data_type: int
description: day monthly number of the given date.
tests:
- not_null
- not_null

14
models/reporting/xero/schema.yaml → models/reporting/xero/schema.yml
View file

@ -798,7 +798,7 @@ models:
tests:
- not_null
- unique
- name: id_prepayment
data_type: character varying
description: |
@ -1072,7 +1072,7 @@ models:
- name: line_amount_wo_taxes_in_gbp
data_type: numeric
description: |
description: |
The total value transferred in this line, minus taxes, presented in
GBP.
@ -1083,12 +1083,12 @@ models:
- name: tax_amount_local_curr
data_type: numeric
description: |
description: |
The total value of taxes for this line, in the transaction currency.
- name: tax_amount_in_gbp
data_type: numeric
description: |
description: |
The total value of taxes for this line, presented in GBP.
- name: tax_type
@ -1151,7 +1151,7 @@ models:
Most fields are not documented here: you can find more details in upstream
models.
columns:
- name: id_line_item
data_type: text
@ -1172,8 +1172,8 @@ models:
- not_null
- accepted_values:
values:
- 'invoice'
- 'credit note'
- "invoice"
- "credit note"
- name: item_code
data_type: text

10
models/staging/core/schema.yml
View file

@ -106,7 +106,7 @@ models:
tests:
- unique
- not_null
- name: stg_core__accommodation
columns:
- name: id_accommodation
@ -195,8 +195,7 @@ models:
- unique
- not_null
- name: stg_core__edeposit_user
description:
"This table contains data on partner users for E-deposit,
description: "This table contains data on partner users for E-deposit,
their currencies and amount of protection according to the
level given on the verification"
columns:
@ -216,8 +215,7 @@ models:
- name: currency
data_type: character varying
description:
"Three-letter ISO code assigned to the currency used by user."
description: "Three-letter ISO code assigned to the currency used by user."
tests:
- not_null
- dbt_expectations.expect_column_values_to_match_regex:
@ -275,4 +273,4 @@ models:
- name: dwh_extracted_at_utc
data_type: timestamp with time zone
description: "Timestamp of when data was extracted to DWH"
description: "Timestamp of when data was extracted to DWH"

32
models/staging/edeposit/schema.yml
View file

@ -2,9 +2,9 @@ version: 2
models:
- name: stg_edeposit__verifications
description:
description:
"Records of each transaction that happens in the edeposit API. Records are
mutable and can get updated."
mutable and can get updated."
columns:
- name: id_verification
data_type: character varying
@ -12,7 +12,7 @@ models:
tests:
- unique
- not_null
- name: id_booking
data_type: text
description: ""
@ -32,15 +32,15 @@ models:
- name: version
data_type: text
description: |
description: |
Indicates whether the verification is for V1 (Athena) or V2
(e-deposit).
tests:
- not_null
- accepted_values:
values:
- 'V1'
- 'V2'
- "V1"
- "V2"
- name: nightly_fee_local
data_type: numeric
@ -74,7 +74,7 @@ models:
- name: email_flag
data_type: text
description: |
description: |
Null if the email shows no issues, otherwise it details the problems
attached to the given email.
tests:
@ -86,7 +86,7 @@ models:
- name: phone_flag
data_type: text
description: |
description: |
Null if the phone number shows no issues, otherwise it details the
problems attached to the given phone number.
tests:
@ -95,7 +95,7 @@ models:
- "Phone number not reachable"
- "Not a real phone number"
- "Phone number is disposable"
- name: watch_list
data_type: text
description: ""
@ -119,7 +119,7 @@ models:
- name: is_cancelled
data_type: boolean
description: |
description: |
Indicates if the booking has been cancelled or not. At the source,
null and false values have the same meaning, so here we turn nulls
into false to keep thing simple.
@ -214,7 +214,7 @@ models:
tests:
- not_null:
where: version = 'V1'
- name: status_updated_date_utc
data_type: timestamp without time zone
description: |
@ -226,15 +226,15 @@ models:
- name: updated_at_utc
data_type: timestamp without time zone
description: |
description: |
Timestamp of the last edit of the record, as set by the
Athena/e-deposit application.
tests:
- not_null
- name: updated_date_utc
data_type: timestamp without time zone
description: |
description: |
Date of the last edit of the record, as set by the
Athena/e-deposit application.
tests:
@ -254,14 +254,14 @@ models:
- name: created_at_utc
data_type: timestamp without time zone
description: |
description: |
The internal application timestamp of when this record was created.
tests:
- not_null
- name: created_date_utc
data_type: timestamp without time zone
description: |
description: |
The internal application date of when this record was created.
tests:
- not_null

8
models/staging/xero/schema.yml
View file

@ -645,7 +645,7 @@ models:
- SPEND-PREPAYMENT
- RECEIVE-TRANSFER
- SPEND-TRANSFER
- name: transaction_sign
data_type: numeric
description: |
@ -747,9 +747,9 @@ models:
- name: full_contact_details
data_type: jsonb
description: |
description: |
The contact related to this transaction.
Might be null since not all transactions have a related contact.
- name: line_items
@ -793,7 +793,7 @@ models:
- name: dwh_extracted_at_utc
data_type: timestamp with time zone
description: ""
- name: stg_xero__accounts
description: |
Accounts in our accounting tree.

2
seeds/schema.yaml → seeds/schema.yml
View file

@ -61,7 +61,7 @@ seeds:
- name: country_name
data_type: character varying
description: The name of the country.
tests:
- not_null
- unique