Merged PR 2689: KPIs by Billing Country

# Description

Adds Billing Country dimension in KPIs, but does not expose them to reporting yet.
Silly thing, based on the macros I built, I cannot make incremental changes unless changing all models. This will need to be adapted, happy to hear your thoughts on how we do it.
Additionally, I have lack of performance of the model `mtd_guest_payments_metrics`. It takes around 5 min to execute, but technically the end-to-end runs in one shoot without breaking.
It's a complex PR because it changes many files, but you will see that:
* It mostly changes the join conditions for the dimensions or the schema tests,
* I tried to be very careful and add things step-by-step in the commits.

Goal is NOT to complete the PR yet until we see how we can improve performance. I can say though that data end-to-end looks ok to me, but would benefit from checking with production data for the new dimension

Update 30th Aug
* Added a new commit that includes `id_user_host` in `int_core__verification_payments`. Happy to discuss if it makes sense or not. But it changes the execution from ~600 sec to ~6 sec because it avoids a massive repeated join with `verification_requests`.

# Checklist

- [X] The edited models and dependants run properly with production data.
- [X] The edited models are sufficiently documented.
- [X] The edited models contain PK tests, and I've ran and passed them.
- [X] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models. **To check because of performance issues**

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #19082

macros/business_kpis_configuration.sql

@@ -9,7 +9,8 @@ Please note that strings should be encoded with " ' your_value_here ' ",
 {% macro get_kpi_dimensions() %}
     {% set dimensions = [
         {"dimension": "'global'", "dimension_value": "'global'"},
-        {"dimension": "'by_number_of_listings'", "dimension_value": "active_accommodations_per_deal_segmentation"}
+        {"dimension": "'by_number_of_listings'", "dimension_value": "active_accommodations_per_deal_segmentation"},
+        {"dimension": "'by_billing_country'", "dimension_value": "billing_country_iso_3"}
     ] %}
     {{ return(dimensions) }}
 {% endmacro %}

models/intermediate/core/int_core__mtd_accommodation_metrics.sql

@@ -96,6 +96,11 @@ with
                     int_core__mtd_accommodation_segmentation mas
                     on u.id_deal = mas.id_deal
                     and al.date = mas.date
+            {% elif dimension.dimension == "'by_billing_country'" %}
+                inner join int_core__unique_accommodation_to_user atu
+                    on atu.id_accommodation = al.id_accommodation
+                inner join int_core__user_host u on atu.id_user_owner = u.id_user_host
+                    and u.billing_country_iso_3 is not null
             {% endif %}
             group by 1, 2, 3
             {% if not loop.last %}

models/intermediate/core/int_core__mtd_booking_metrics.sql

@@ -36,6 +36,9 @@ with
                 inner join int_core__mtd_accommodation_segmentation mas
                     on u.id_deal = mas.id_deal
                     and d.date = mas.date
+            {% elif dimension.dimension == "'by_billing_country'" %}
+                inner join int_core__user_host u on b.id_user_host = u.id_user_host
+                    and u.billing_country_iso_3 is not null
             {% endif %}
             group by 1, 2, 3
             {% if not loop.last %}
@@ -60,6 +63,9 @@ with
                 inner join int_core__mtd_accommodation_segmentation mas
                     on u.id_deal = mas.id_deal
                     and d.date = mas.date
+            {% elif dimension.dimension == "'by_billing_country'" %}
+                inner join int_core__user_host u on b.id_user_host = u.id_user_host
+                    and u.billing_country_iso_3 is not null
             {% endif %}
             group by 1, 2, 3
             {% if not loop.last %}
@@ -85,6 +91,9 @@ with
                 inner join int_core__mtd_accommodation_segmentation mas
                     on u.id_deal = mas.id_deal
                     and d.date = mas.date
+            {% elif dimension.dimension == "'by_billing_country'" %}
+                inner join int_core__user_host u on b.id_user_host = u.id_user_host
+                    and u.billing_country_iso_3 is not null
             {% endif %}
             group by 1, 2, 3
             {% if not loop.last %}
@@ -111,6 +120,10 @@ with
                     int_core__mtd_accommodation_segmentation mas
                     on u.id_deal = mas.id_deal
                     and d.date = mas.date
+            {% elif dimension.dimension == "'by_billing_country'" %}
+                inner join int_core__bookings b on b.id_booking = bce.id_booking
+                inner join int_core__user_host u on b.id_user_host = u.id_user_host
+                    and u.billing_country_iso_3 is not null
             {% endif %}
             group by 1, 2, 3
             {% if not loop.last %}

models/intermediate/core/int_core__mtd_deal_metrics.sql

@@ -17,7 +17,9 @@ with
     int_core__mtd_accommodation_segmentation as (
         select * from {{ ref("int_core__mtd_accommodation_segmentation") }}
     ),
-
+    int_core__user_host as (
+        select * from {{ ref("int_core__user_host") }}
+    ),
     deals_metric_aggregation_per_date as (
         {% for dimension in dimensions %}
             select
@@ -88,6 +90,9 @@ with
                     int_core__mtd_accommodation_segmentation mas
                     on al.id_deal = mas.id_deal
                     and al.date = mas.date
+            {% elif dimension.dimension == "'by_billing_country'" %}
+                inner join int_core__user_host u on al.id_deal = u.id_deal
+                    and u.billing_country_iso_3 is not null
             {% endif %}
             group by 1, 2, 3
             {% if not loop.last %}

models/intermediate/core/int_core__mtd_guest_journey_metrics.sql

@@ -49,6 +49,10 @@ with
                 inner join int_core__mtd_accommodation_segmentation mas
                     on u.id_deal = mas.id_deal
                     and d.date = mas.date
+            {% elif dimension.dimension == "'by_billing_country'" %}
+                inner join int_core__user_host u 
+                    on vr.id_user_host = u.id_user_host
+                    and u.billing_country_iso_3 is not null
             {% endif %}
             group by 1, 2, 3
             {% if not loop.last %}
@@ -74,6 +78,10 @@ with
                 inner join int_core__mtd_accommodation_segmentation mas
                     on u.id_deal = mas.id_deal
                     and d.date = mas.date
+            {% elif dimension.dimension == "'by_billing_country'" %}
+                inner join int_core__user_host u 
+                    on vr.id_user_host = u.id_user_host
+                    and u.billing_country_iso_3 is not null
             {% endif %}
             group by 1, 2, 3
             {% if not loop.last %}
@@ -99,6 +107,10 @@ with
                 inner join int_core__mtd_accommodation_segmentation mas
                     on u.id_deal = mas.id_deal
                     and d.date = mas.date
+            {% elif dimension.dimension == "'by_billing_country'" %}
+                inner join int_core__user_host u 
+                    on vr.id_user_host = u.id_user_host
+                    and u.billing_country_iso_3 is not null
             {% endif %}
             group by 1, 2, 3
             {% if not loop.last %}
@@ -125,6 +137,11 @@ with
                 inner join int_core__mtd_accommodation_segmentation mas
                     on u.id_deal = mas.id_deal
                     and d.date = mas.date
+            {% elif dimension.dimension == "'by_billing_country'" %}
+                inner join int_core__verification_requests vr on vr.id_verification_request = p.id_verification_request
+                inner join int_core__user_host u 
+                    on vr.id_user_host = u.id_user_host
+                    and u.billing_country_iso_3 is not null
             {% endif %}
             group by 1, 2, 3
             {% if not loop.last %}

models/intermediate/core/int_core__mtd_guest_payments_metrics.sql

@@ -65,14 +65,14 @@ with
                 on date_trunc('month', vp.payment_paid_date_utc)::date = d.first_day_month
                 and extract(day from vp.payment_paid_date_utc) <= d.day
             {% if dimension.dimension == "'by_number_of_listings'" %}
-                inner join
+                inner join int_core__user_host u on vp.id_user_host = u.id_user_host
-                    int_core__verification_requests vr
-                    on vp.id_verification_request = vr.id_verification_request
-                inner join int_core__user_host u on vr.id_user_host = u.id_user_host
                 inner join
                     int_core__mtd_accommodation_segmentation mas
                     on u.id_deal = mas.id_deal
                     and d.date = mas.date
+            {% elif dimension.dimension == "'by_billing_country'" %}
+                inner join int_core__user_host u on vp.id_user_host = u.id_user_host
+                    and u.billing_country_iso_3 is not null
             {% endif %}
             where upper(vp.payment_status) = {{ var("paid_payment_state") }}
             group by 1, 2, 3

models/intermediate/core/int_core__verification_payments.sql

@@ -7,6 +7,7 @@ with
         select * from {{ ref("stg_core__verification_payment_type") }}
     ),
     stg_core__verification as (select * from {{ ref("stg_core__verification") }}),
+    stg_core__verification_request as (select * from {{ ref("stg_core__verification_request") }}),
     stg_core__payment as (select * from {{ ref("stg_core__payment") }}),
     stg_core__payment_status as (select * from {{ ref("stg_core__payment_status") }}),
     int_simple_exchange_rates as (select * from {{ ref("int_simple_exchange_rates") }})
@@ -26,6 +27,10 @@ select
     p.refunded_at_utc as payment_refunded_at_utc,
     p.refunded_date_utc as payment_refunded_date_utc,
     p.refund_payment_reference,
+    -- Host User identifier is included to speed up
+    -- KPIs execution, even though the host itself
+    -- has nothing to do with the guest payments.
+    vr.id_user_host,
     vtp.id_guest_user,
     vtp.id_verification,
     v.id_verification_request,
@@ -47,3 +52,6 @@ left join
     on vtp.payment_due_date_utc = r.rate_date_utc
     and p.currency = r.from_currency
     and r.to_currency = 'GBP'
+left join 
+    stg_core__verification_request vr 
+    on v.id_verification_request = vr.id_verification_request

models/intermediate/core/schema.yaml

@@ -542,6 +542,7 @@ models:
                 values:
                 - global
                 - by_number_of_listings
+                - by_billing_country
       
       - name: dimension_value
         data_type: string
@@ -577,6 +578,7 @@ models:
                 values:
                 - global
                 - by_number_of_listings
+                - by_billing_country
       
       - name: dimension_value
         data_type: string
@@ -612,6 +614,7 @@ models:
                 values:
                 - global
                 - by_number_of_listings
+                - by_billing_country
       
       - name: dimension_value
         data_type: string
@@ -647,6 +650,7 @@ models:
                 values:
                 - global
                 - by_number_of_listings
+                - by_billing_country
       
       - name: dimension_value
         data_type: string
@@ -682,6 +686,7 @@ models:
                 values:
                 - global
                 - by_number_of_listings
+                - by_billing_country
       
       - name: dimension_value
         data_type: string
@@ -870,6 +875,13 @@ models:
         data_type: date
       - name: refund_payment_reference
         data_type: character varying
+      - name: id_user_host
+        data_type: character varying
+        description: |
+          UUID of the Host linked to the Verification Request
+          that has a payment. It's included here to speed up
+          KPIs execution, even though the host itself has nothing 
+          to do with the guest payments.
       - name: id_guest_user
         data_type: character varying
       - name: id_verification

models/intermediate/cross/int_dates_mtd_by_dimension.sql

@@ -7,6 +7,9 @@ with
     int_core__mtd_accommodation_segmentation as (
         select * from {{ ref("int_core__mtd_accommodation_segmentation") }}
     ),
+    int_core__user_host as (
+        select * from {{ ref("int_core__user_host") }}
+    ),
     int_dates_mtd as (select * from {{ ref("int_dates_mtd") }})
 
 {% for dimension in dimensions %}
@@ -25,6 +28,10 @@ with
     {% if dimension.dimension == "'by_number_of_listings'" %}
         inner join int_core__mtd_accommodation_segmentation a
 		on d.date = a.date
+    {% elif dimension.dimension == "'by_billing_country'" %}
+        inner join int_core__user_host h 
+        on d.date >= h.created_date_utc 
+        and h.billing_country_iso_3 is not null
     {% endif %} 
     {% if not loop.last %}
         union all

models/intermediate/cross/schema.yml

@@ -167,6 +167,7 @@ models:
                 values:
                 - global
                 - by_number_of_listings
+                - by_billing_country
       
       - name: dimension_value
         data_type: string
@@ -375,6 +376,7 @@ models:
                 values:
                 - global
                 - by_number_of_listings
+                - by_billing_country
       
       - name: dimension_value
         data_type: string
@@ -549,6 +551,7 @@ models:
                 values:
                 - global
                 - by_number_of_listings
+                - by_billing_country
       
       - name: dimension_value
         data_type: string

models/intermediate/xero/int_xero__mtd_invoicing_metrics.sql

@@ -34,6 +34,7 @@ with
         select * from {{ ref("int_dates_mtd_by_dimension") }}
     ),
     int_xero__contacts as (select * from {{ ref("int_xero__contacts") }}),
+    int_core__user_host as (select * from {{ ref("int_core__user_host") }}),
     resolution_host_payment as (
         {% for dimension in dimensions %}
             select
@@ -62,6 +63,12 @@ with
                     int_core__mtd_accommodation_segmentation mas
                     on c.id_deal = mas.id_deal
                     and d.date = mas.date
+            {% elif dimension.dimension == "'by_billing_country'" %}
+                inner join int_xero__contacts c on c.id_contact = bt.id_contact
+                inner join
+                    int_core__user_host u
+                    on c.id_deal = u.id_deal
+                    and u.billing_country_iso_3 is not null
             {% endif %}
             group by 1, 2, 3
             {% if not loop.last %}
@@ -99,6 +106,11 @@ with
                     int_core__mtd_accommodation_segmentation mas
                     on sdm.id_deal = mas.id_deal
                     and d.date = mas.date
+            {% elif dimension.dimension == "'by_billing_country'" %}
+                inner join
+                    int_core__user_host u
+                    on sdm.id_deal = u.id_deal
+                    and u.billing_country_iso_3 is not null
             {% endif %}
             where
                 upper(sdm.document_status) in {{ relevant_document_statuses }}
@@ -156,6 +168,11 @@ with
                     int_core__mtd_accommodation_segmentation mas
                     on sdm.id_deal = mas.id_deal
                     and d.date = mas.date
+            {% elif dimension.dimension == "'by_billing_country'" %}
+                inner join
+                    int_core__user_host u
+                    on sdm.id_deal = u.id_deal
+                    and u.billing_country_iso_3 is not null
             {% endif %}
             where
                 upper(sdm.document_status) in {{ relevant_document_statuses }}

models/intermediate/xero/schema.yaml

@@ -313,6 +313,7 @@ models:
                 values:
                 - global
                 - by_number_of_listings
+                - by_billing_country
       
       - name: dimension_value
         data_type: string