Merged PR 4387: Adding booking services status

# Description

Need opinion on this model, it's not so easy so if needed we can have a quick call
- Since the `int_core__booking_service_detail` has some services combined in one I tried to split them to count separately each service per booking and their statuses. There are also some cases where there are services combined but also separate for the same booking, in those cases I only take in count the status of the service with the most recent `service_detail_updated_at_utc `
![image (2).png](https://guardhog.visualstudio.com/4148d95f-4b6d-4205-bcff-e9c8e0d2ca65/_apis/git/repositories/54ac356f-aad7-46d2-b62c-e8c5b3bb8ebf/pullRequests/4387/attachments/image%20%282%29.png)
So here it takes for the `BASIC DAMAGE DEPOSIT` the status Paid which is more recent but for `WAIVER PLUS` the status Not Paid.
- I created a macro to count all the distinct bookings with each `service_status` that also adds the column name. If there are new statuses being used we would have to add them to the macro as well as calling the new columns on the final select. Might not be the most optimal approach WDYT??

**The PR is not ready to be closed**

# Checklist

- [x] The edited models and dependants run properly with production data.
- [ ] The edited models are sufficiently documented.
- [ ] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Adding booking services status

Related work items: #25934

dbt_project.yml

@@ -75,11 +75,6 @@ vars:
   # States should be strings in capital letters. Models need to force an upper()
   "cancelled_booking_state": "'CANCELLED'"
   "approved_booking_state": "'APPROVED'"
-  "flagged_booking_state": "'FLAGGED'"
-  "incomplete_information_booking_state": "'INCOMPLETEINFORMATION'"
-  "no_flags_booking_state": "'NOFLAGS'"
-  "not_approved_booking_state": "'NOTAPPROVED'"
-  "rejected_booking_state": "'REJECTED'"
 
   # Payment state variables
   # States should be strings in capital letters. Models need to force an upper()

models/intermediate/core/int_core__new_dash_services_offered.sql

@@ -1,3 +1,17 @@
+{% set protected_service_status = "'PROTECTED'" %}
+{% set rejected_service_status = "'REJECTED'" %}
+{% set no_checks_service_status = "'NOCHECKS'" %}
+{% set no_flags_service_status = "'NOFLAGS'" %}
+{% set paid_service_status = "'PAID'" %}
+{% set pending_service_status = "'PENDING'" %}
+{% set unknown_service_status = "'-'" %}
+{% set partially_protected_service_status = "'PARTIALLY PROTECTED'" %}
+{% set not_protected_service_status = "'NOT PROTECTED'" %}
+{% set not_paid_service_status = "'NOT PAID'" %}
+{% set confirmed_service_status = "'CONFIRMED'" %}
+{% set for_review_service_status = "'FORREVIEW'" %}
+{% set flagged_service_status = "'FLAGGED'" %}
+
 with
     int_core__user_product_bundle_contains_services as (
         select * from {{ ref("int_core__user_product_bundle_contains_services") }}
@@ -13,7 +27,9 @@ with
     int_core__booking_to_product_bundle as (
         select * from {{ ref("int_core__booking_to_product_bundle") }}
     ),
-    int_core__bookings as (select * from {{ ref("int_core__bookings") }}),
+    int_core__booking_service_detail as (
+        select * from {{ ref("int_core__booking_service_detail") }}
+    ),
     bundle_services as (
         select
             bs.id_user_product_bundle,
@@ -45,7 +61,7 @@ with
             and uh.is_test_account = false
     ),
     users as (
-        select distinct
+        select
             bs.service_display_name,
             count(distinct bs.id_user_host) as number_users,
             -- Count only users that have at least one active accommodation.
@@ -66,7 +82,7 @@ with
         group by 1
     ),
     accommodations as (
-        select distinct
+        select
             bs.service_display_name,
             count(distinct apb.id_accommodation) as number_accommodations,
             count(
@@ -88,73 +104,107 @@ with
         group by 1
     ),
     bookings as (
-        select distinct
+        select
-            bs.service_display_name,
+            sd.service_name as service_display_name,
-            count(distinct bpb.id_booking) as number_bookings,
+            count(distinct sd.id_booking) as number_bookings,
             count(
                 distinct case
-                    when upper(b.booking_state) = {{ var("approved_booking_state") }}
+                    when upper(sd.service_status) = {{ protected_service_status }}
-                    then bpb.id_booking
+                    then sd.id_booking
                     else null
                 end
-            ) as number_approved_bookings,
+            ) as number_bookings_with_service_status_protected,
             count(
                 distinct case
-                    when upper(b.booking_state) = {{ var("cancelled_booking_state") }}
+                    when upper(sd.service_status) = {{ rejected_service_status }}
-                    then bpb.id_booking
+                    then sd.id_booking
                     else null
                 end
-            ) as number_cancelled_bookings,
+            ) as number_bookings_with_service_status_rejected,
             count(
                 distinct case
-                    when upper(b.booking_state) = {{ var("flagged_booking_state") }}
+                    when upper(sd.service_status) = {{ no_checks_service_status }}
-                    then bpb.id_booking
+                    then sd.id_booking
                     else null
                 end
-            ) as number_flagged_bookings,
+            ) as number_bookings_with_service_status_no_checks,
+            count(
+                distinct case
+                    when upper(sd.service_status) = {{ no_flags_service_status }}
+                    then sd.id_booking
+                    else null
+                end
+            ) as number_bookings_with_service_status_no_flags,
+            count(
+                distinct case
+                    when upper(sd.service_status) = {{ paid_service_status }}
+                    then sd.id_booking
+                    else null
+                end
+            ) as number_bookings_with_service_status_paid,
+            count(
+                distinct case
+                    when upper(sd.service_status) = {{ paid_service_status }}
+                    then sd.id_booking
+                    else null
+                end
+            ) as number_bookings_with_service_status_pending,
+            count(
+                distinct case
+                    when upper(sd.service_status) = {{ unknown_service_status }}
+                    then sd.id_booking
+                    else null
+                end
+            ) as number_bookings_with_service_status_unknown_status,
             count(
                 distinct case
                     when
-                        upper(b.booking_state)
+                        upper(sd.service_status)
-                        = {{ var("incomplete_information_booking_state") }}
+                        = {{ partially_protected_service_status }}
-                    then bpb.id_booking
+                    then sd.id_booking
                     else null
                 end
-            ) as number_incomplete_information_bookings,
+            ) as number_bookings_with_service_status_partially_protected,
             count(
                 distinct case
-                    when upper(b.booking_state) = {{ var("no_flags_booking_state") }}
+                    when upper(sd.service_status) = {{ not_protected_service_status }}
-                    then bpb.id_booking
+                    then sd.id_booking
                     else null
                 end
-            ) as number_no_flags_bookings,
+            ) as number_bookings_with_service_status_not_protected,
             count(
                 distinct case
-                    when
+                    when upper(sd.service_status) = {{ not_paid_service_status }}
-                        upper(b.booking_state) = {{ var("not_approved_booking_state") }}
+                    then sd.id_booking
-                    then bpb.id_booking
                     else null
                 end
-            ) as number_not_approved_bookings,
+            ) as number_bookings_with_service_status_not_paid,
             count(
                 distinct case
-                    when upper(b.booking_state) = {{ var("rejected_booking_state") }}
+                    when upper(sd.service_status) = {{ confirmed_service_status }}
-                    then bpb.id_booking
+                    then sd.id_booking
                     else null
                 end
-            ) as number_rejected_bookings
+            ) as number_bookings_with_service_status_confirmed,
-        from bundle_services bs
+            count(
-        left join
+                distinct case
-            int_core__booking_to_product_bundle bpb
+                    when upper(sd.service_status) = {{ for_review_service_status }}
-            on bpb.id_user_product_bundle = bs.id_user_product_bundle
+                    then sd.id_booking
-        inner join
+                    else null
-            int_core__bookings b
+                end
-            on b.id_booking = bpb.id_booking
+            ) as number_bookings_with_service_status_for_review,
-            and b.is_duplicate_booking is false
+            count(
+                distinct case
+                    when upper(sd.service_status) = {{ flagged_service_status }}
+                    then sd.id_booking
+                    else null
+                end
+            ) as number_bookings_with_service_status_flagged
+        from int_core__booking_service_detail sd
         group by 1
     )
 select
-    u.service_display_name,
+    coalesce(u.service_display_name, b.service_display_name) as service_display_name,
     u.number_users,
     u.number_users_with_service_applied_in_accommodation,
     u.number_users
@@ -164,13 +214,19 @@ select
     a.number_active_accommodations,
     a.number_inactive_accommodations,
     b.number_bookings,
-    b.number_approved_bookings,
+    b.number_bookings_with_service_status_protected,
-    b.number_cancelled_bookings,
+    b.number_bookings_with_service_status_rejected,
-    b.number_flagged_bookings,
+    b.number_bookings_with_service_status_no_checks,
-    b.number_incomplete_information_bookings,
+    b.number_bookings_with_service_status_no_flags,
-    b.number_no_flags_bookings,
+    b.number_bookings_with_service_status_paid,
-    b.number_not_approved_bookings,
+    b.number_bookings_with_service_status_pending,
-    b.number_rejected_bookings
+    b.number_bookings_with_service_status_unknown_status,
+    b.number_bookings_with_service_status_partially_protected,
+    b.number_bookings_with_service_status_not_protected,
+    b.number_bookings_with_service_status_not_paid,
+    b.number_bookings_with_service_status_confirmed,
+    b.number_bookings_with_service_status_for_review,
+    b.number_bookings_with_service_status_flagged
 from users u
 left join accommodations a on u.service_display_name = a.service_display_name
-left join bookings b on u.service_display_name = b.service_display_name
+full outer join bookings b on u.service_display_name = b.service_display_name

models/intermediate/core/schema.yml

@@ -5241,19 +5241,6 @@ models:
         description: "The name of the New Dash service."
         data_tests:
           - not_null
-          - accepted_values:
-              values:
-                - "BASIC SCREENING"
-                - "SCREENING PLUS"
-                - "ID VERIFICATION"
-                - "SEX OFFENDERS CHECK"
-                - "BASIC DAMAGE DEPOSIT"
-                - "BASIC WAIVER"
-                - "WAIVER PLUS"
-                - "WAIVER PRO"
-                - "BASIC PROTECTION"
-                - "PROTECTION PLUS"
-                - "PROTECTION PRO"
 
       - name: number_users
         data_type: bigint
@@ -5302,47 +5289,57 @@ models:
         data_type: bigint
         description: "Number of bookings that have a bundle that considers this service."
 
-      - name: number_approved_bookings
+      - name: number_bookings_with_service_status_protected
         data_type: bigint
-        description:
+        description: "Number of bookings with status PROTECTED for this service."
-          "Number of bookings that have a bundle that considers this service and
-          are approved."
 
-      - name: number_cancelled_bookings
+      - name: number_bookings_with_service_status_rejected
         data_type: bigint
-        description:
+        description: "Number of bookings with status REJECTED for this service."
-          "Number of bookings that have a bundle that considers this service and
-          are cancelled."
 
-      - name: number_flagged_bookings
+      - name: number_bookings_with_service_status_no_checks
         data_type: bigint
-        description:
+        description: "Number of bookings with status NO CHECKS for this service."
-          "Number of bookings that have a bundle that considers this service and
-          are flagged."
 
-      - name: number_incomplete_information_bookings
+      - name: number_bookings_with_service_status_no_flags
         data_type: bigint
-        description:
+        description: "Number of bookings with status NO FLAGS for this service."
-          "Number of bookings that have a bundle that considers this service and
-          have incomplete information."
 
-      - name: number_no_flags_bookings
+      - name: number_bookings_with_service_status_paid
         data_type: bigint
-        description:
+        description: "Number of bookings with status PAID for this service."
-          "Number of bookings that have a bundle that considers this service and
-          have no flags."
 
-      - name: number_not_approved_bookings
+      - name: number_bookings_with_service_status_pending
         data_type: bigint
-        description:
+        description: "Number of bookings with status PENDING for this service."
-          "Number of bookings that have a bundle that considers this service and
-          are not approved."
 
-      - name: number_rejected_bookings
+      - name: number_bookings_with_service_status_unknown
         data_type: bigint
-        description:
+        description: "Number of bookings with unknown status for this service."
-          "Number of bookings that have a bundle that considers this service and
+
-          are rejected."
+      - name: number_bookings_with_service_status_partially_protected
+        data_type: bigint
+        description: "Number of bookings with status PARTIALLY PROTECTED for this service."
+
+      - name: number_bookings_with_service_status_not_protected
+        data_type: bigint
+        description: "Number of bookings with status NOT PROTECTED for this service."
+
+      - name: number_bookings_with_service_status_not_paid
+        data_type: bigint
+        description: "Number of bookings with status NOT PAID for this service."
+
+      - name: number_bookings_with_service_status_confirmed
+        data_type: bigint
+        description: "Number of bookings with status CONFIRMED for this service."
+
+      - name: number_bookings_with_service_status_for_review
+        data_type: bigint
+        description: "Number of bookings with status FOR REVIEW for this service."
+
+      - name: number_bookings_with_service_status_flagged
+        data_type: bigint
+        description: "Number of bookings with status FLAGGED for this service."
 
   - name: int_core__payments
     description: |

models/reporting/core/core__new_dash_services_offered.sql

@@ -13,11 +13,30 @@ select
     number_active_accommodations as number_active_accommodations,
     number_inactive_accommodations as number_inactive_accommodations,
     number_bookings as number_bookings,
-    number_approved_bookings as number_approved_bookings,
+    number_bookings_with_service_status_protected
-    number_cancelled_bookings as number_cancelled_bookings,
+    as number_bookings_with_service_status_protected,
-    number_flagged_bookings as number_flagged_bookings,
+    number_bookings_with_service_status_rejected
-    number_incomplete_information_bookings as number_incomplete_information_bookings,
+    as number_bookings_with_service_status_rejected,
-    number_no_flags_bookings as number_no_flags_bookings,
+    number_bookings_with_service_status_no_checks
-    number_not_approved_bookings as number_not_approved_bookings,
+    as number_bookings_with_service_status_no_checks,
-    number_rejected_bookings as number_rejected_bookings
+    number_bookings_with_service_status_no_flags
+    as number_bookings_with_service_status_no_flags,
+    number_bookings_with_service_status_paid
+    as number_bookings_with_service_status_paid,
+    number_bookings_with_service_status_pending
+    as number_bookings_with_service_status_pending,
+    number_bookings_with_service_status_unknown_status
+    as number_bookings_with_service_status_unknown_status,
+    number_bookings_with_service_status_partially_protected
+    as number_bookings_with_service_status_partially_protected,
+    number_bookings_with_service_status_not_protected
+    as number_bookings_with_service_status_not_protected,
+    number_bookings_with_service_status_not_paid
+    as number_bookings_with_service_status_not_paid,
+    number_bookings_with_service_status_confirmed
+    as number_bookings_with_service_status_confirmed,
+    number_bookings_with_service_status_for_review
+    as number_bookings_with_service_status_for_review,
+    number_bookings_with_service_status_flagged
+    as number_bookings_with_service_status_flagged
 from int_core__new_dash_services_offered

models/reporting/core/schema.yml

@@ -1579,19 +1579,6 @@ models:
         description: "The name of the New Dash service."
         data_tests:
           - not_null
-          - accepted_values:
-              values:
-                - "BASIC SCREENING"
-                - "SCREENING PLUS"
-                - "ID VERIFICATION"
-                - "SEX OFFENDERS CHECK"
-                - "BASIC DAMAGE DEPOSIT"
-                - "BASIC WAIVER"
-                - "WAIVER PLUS"
-                - "WAIVER PRO"
-                - "BASIC PROTECTION"
-                - "PROTECTION PLUS"
-                - "PROTECTION PRO"
 
       - name: number_users
         data_type: bigint
@@ -1640,47 +1627,57 @@ models:
         data_type: bigint
         description: "Number of bookings that have a bundle that considers this service."
 
-      - name: number_approved_bookings
+      - name: number_bookings_with_service_status_protected
         data_type: bigint
-        description:
+        description: "Number of bookings with status PROTECTED for this service."
-          "Number of bookings that have a bundle that considers this service and
-          are approved."
 
-      - name: number_cancelled_bookings
+      - name: number_bookings_with_service_status_rejected
         data_type: bigint
-        description:
+        description: "Number of bookings with status REJECTED for this service."
-          "Number of bookings that have a bundle that considers this service and
-          are cancelled."
 
-      - name: number_flagged_bookings
+      - name: number_bookings_with_service_status_no_checks
         data_type: bigint
-        description:
+        description: "Number of bookings with status NO CHECKS for this service."
-          "Number of bookings that have a bundle that considers this service and
-          are flagged."
 
-      - name: number_incomplete_information_bookings
+      - name: number_bookings_with_service_status_no_flags
         data_type: bigint
-        description:
+        description: "Number of bookings with status NO FLAGS for this service."
-          "Number of bookings that have a bundle that considers this service and
-          have incomplete information."
 
-      - name: number_no_flags_bookings
+      - name: number_bookings_with_service_status_paid
         data_type: bigint
-        description:
+        description: "Number of bookings with status PAID for this service."
-          "Number of bookings that have a bundle that considers this service and
-          have no flags."
 
-      - name: number_not_approved_bookings
+      - name: number_bookings_with_service_status_pending
         data_type: bigint
-        description:
+        description: "Number of bookings with status PENDING for this service."
-          "Number of bookings that have a bundle that considers this service and
-          are not approved."
 
-      - name: number_rejected_bookings
+      - name: number_bookings_with_service_status_unknown
         data_type: bigint
-        description:
+        description: "Number of bookings with unknown status for this service."
-          "Number of bookings that have a bundle that considers this service and
+
-          are rejected."
+      - name: number_bookings_with_service_status_partially_protected
+        data_type: bigint
+        description: "Number of bookings with status PARTIALLY PROTECTED for this service."
+
+      - name: number_bookings_with_service_status_not_protected
+        data_type: bigint
+        description: "Number of bookings with status NOT PROTECTED for this service."
+
+      - name: number_bookings_with_service_status_not_paid
+        data_type: bigint
+        description: "Number of bookings with status NOT PAID for this service."
+
+      - name: number_bookings_with_service_status_confirmed
+        data_type: bigint
+        description: "Number of bookings with status CONFIRMED for this service."
+
+      - name: number_bookings_with_service_status_for_review
+        data_type: bigint
+        description: "Number of bookings with status FOR REVIEW for this service."
+
+      - name: number_bookings_with_service_status_flagged
+        data_type: bigint
+        description: "Number of bookings with status FLAGGED for this service."
 
   - name: core__payments
     description: |