Merged PR 4693: Live Dates according to HubSpot

# Description

Made the change as discussed during the daily.
Here we can see an example of the impact with the change (left old lifecycle, right new one)
The biggest impact comes from the Never Booked deals, 110, and Active 34.
![image.png](https://guardhog.visualstudio.com/4148d95f-4b6d-4205-bcff-e9c8e0d2ca65/_apis/git/repositories/54ac356f-aad7-46d2-b62c-e8c5b3bb8ebf/pullRequests/4693/attachments/image.png)

# Checklist

- [x] The edited models and dependants run properly with production data.
- [x] The edited models are sufficiently documented.
- [x] The edited models contain PK tests, and I've ran and passed them.
- [ ] I have checked for DRY opportunities with other models and docs.
- [ ] I've picked the right materialization for the affected models.

# Other

- [ ] Check if a full-refresh is required after this PR is merged.

Related work items: #28475

models/intermediate/kpis/int_kpis__dimension_deals.sql

@@ -38,6 +38,9 @@ with
         select
             coalesce(hd.id_deal, cd.id_deal) as id_deal,
             coalesce(hd.main_deal_name, cd.main_deal_name) as main_deal_name,
+            case
+                when hd.id_deal is not null then true else false
+            end as is_deal_in_hubspot,
             coalesce(cd.has_active_pms, false) as has_active_pms,
             cd.active_pms_list,
             cd.main_billing_country_iso_3_per_deal,
@@ -55,11 +58,12 @@ with
             min(hd.hubspot_listing_segmentation) as hubspot_listing_segmentation
         from hubspot_deals hd
         full outer join core_deals cd on hd.id_deal = cd.id_deal
-        group by 1, 2, 3, 4, 5, 6
+        group by 1, 2, 3, 4, 5, 6, 7
     )
 select
     cd.id_deal,
     cd.main_deal_name,
+    cd.is_deal_in_hubspot,
     cd.has_active_pms,
     cd.active_pms_list,
     cd.client_type,

models/intermediate/kpis/int_kpis__lifecycle_daily_deal.sql

@@ -46,6 +46,7 @@ with
         select
             ikdd.id_deal,
             ikdd.client_type,
+            ikdd.is_deal_in_hubspot,
             ikdd.effective_deal_start_date_utc as created_date_utc
         from int_kpis__dimension_deals ikdd
     ),
@@ -53,6 +54,7 @@ with
         select
             d.date,
             ikdd.id_deal,
+            ikdd.is_deal_in_hubspot,
             min(ikdd.client_type) as client_type,
             min(ikdd.created_date_utc) as creation_date_utc,
             min(b.created_date_utc) as first_time_booked_date_utc,
@@ -64,12 +66,13 @@ with
             booked_days_per_deal b
             on ikdd.id_deal = b.id_deal
             and d.date >= b.created_date_utc
-        group by d.date, ikdd.id_deal
+        group by d.date, ikdd.id_deal, ikdd.is_deal_in_hubspot
     ),
     deal_historic_features as (
         select
             hhbf.date,
             hhbf.id_deal,
+            hhbf.is_deal_in_hubspot,
             hhbf.creation_date_utc,
             hhbf.first_time_booked_date_utc,
             hhbf.last_time_booked_date_utc,
@@ -157,7 +160,10 @@ select
     case
         -- 01-New: The deal has been created this month.
         -- Additionally, the deal has not been offboarded in hubspot.
-        when deal_was_created_this_month and not deal_has_been_offboarded
+        when
+            deal_was_created_this_month
+            and not deal_has_been_offboarded
+            and is_deal_in_hubspot
         then '01-New'
         -- 02-Never Booked: The deal is not API, has been created before this month
         -- and has not had any booking. Additionally, the deal has not been offboarded
@@ -167,6 +173,7 @@ select
             and not deal_was_created_this_month
             and not deal_has_been_offboarded
             and not is_api_deal
+            and is_deal_in_hubspot
         then '02-Never Booked'
         -- 04-Active: 
         -- The deal is API, is not New and has not been offboarded
@@ -177,6 +184,7 @@ select
             is_api_deal
             and not deal_was_created_this_month
             and not deal_has_been_offboarded
+            and is_deal_in_hubspot
             -- Platform deals --
             or (
                 not is_api_deal
@@ -189,6 +197,7 @@ select
                     had_previous_booking_more_than_12_months_before_the_last
                     and has_been_booked_within_current_month
                 )
+                and is_deal_in_hubspot
             )
         then '04-Active'
         -- 05-Churning: The deal has been offboarded this month.
@@ -204,6 +213,7 @@ select
                 and not is_api_deal
             )
             or deal_was_offboarded_this_month
+            and is_deal_in_hubspot
         then '05-Churning'
         -- 06-Inactive: The deal has been offboarded in the past but not this
         -- month.
@@ -219,6 +229,7 @@ select
                 and not is_api_deal
             )
             or (deal_has_been_offboarded and not deal_was_offboarded_this_month)
+            and is_deal_in_hubspot
         then '06-Inactive'
         -- 07-Reactivated: The deal is not offboarded but was
         -- churned/inactive, and
@@ -228,8 +239,9 @@ select
             and has_been_booked_within_current_month
             and not deal_has_been_offboarded
             and not is_api_deal
+            and is_deal_in_hubspot
         then '07-Reactivated'
-        else null
+        else '99-Not in HubSpot'
     end as deal_lifecycle_state,
     has_been_booked_within_current_month,
     has_been_booked_within_last_6_months,

models/intermediate/kpis/schema.yml

@@ -316,7 +316,7 @@ models:
       - 05-Churning: Either Deals that are offboarded in that month or Deals that are becoming inactive because of lack of bookings in the past 12 months
       - 06-Inactive: Either Deals that have been previously offboarded or Deals that have not had a booking for more than 12 months. 
       - 07-Reactivated: Deals that have had a booking in the current month that were inactive or churning before, that are not offboarded.
-      - Finally, if none of the logic applies, which should not happen, null will be set and a dbt alert will raise.
+      - 99-Not in HubSpot: Deals that are not in HubSpot so we can't determine the lifecycle state.
 
       Since the states of Active, First Time Booked and Reactivated indicate certain booking activity and are
       mutually exclusive, the model also provides information of the recency of the bookings by the following 
@@ -388,7 +388,7 @@ models:
         description: |
           Contains the lifecycle state of a deal. The accepted values are:
           01-New, 02-Never Booked, 04-Active, 05-Churning, 06-Inactive,
-          07-Reactivated. Failing to implement the logic will result in alert.
+          07-Reactivated, 99-Not in Husbpot.
         data_tests:
           - not_null
           - accepted_values:
@@ -399,6 +399,7 @@ models:
                 - 05-Churning
                 - 06-Inactive
                 - 07-Reactivated
+                - 99-Not in HubSpot
 
       - name: has_been_booked_within_current_month
         data_type: boolean
@@ -449,6 +450,10 @@ models:
           in both systems, Hubspot data will take precedence in terms of deal name.
         data_tests:
           - not_null
+      - name: is_deal_in_hubspot
+        data_type: boolean
+        description: |
+          Does the deal exist in HubSpot.
       - name: has_active_pms
         data_type: boolean
         description: |
@@ -479,17 +484,16 @@ models:
       - name: effective_deal_start_date_utc
         data_type: date
         description: |
-          Effective start date of the deal, which corresponds to the minimum between the 
+          Effective start date of the deal, this corresponds to the date a deal has 
-          date a deal has gone live according to Hubspot and the first date a user 
+          gone live according to Hubspot.
-          host has been created according to Core.
         data_tests:
           - not_null
       - name: effective_deal_start_month
         data_type: date
         description: |
-          This field represents the first day of the month of the effective 
+          This field represents the first day of the month of the effective deal 
-          start date of the deal. This is obtained by truncating the effective
+          start date. This is obtained by truncating the effective deal start date 
-          deal start date to the month.
+          to the month.
         data_tests:
           - not_null
       - name: hubspot_deal_cancellation_date_utc