sh-notion/notion_data_team_no_files/Retrieving New Dash MVP info 37429e2b559e492a881c088bdba5ad80.md

Retrieving New Dash MVP info

Screenshots and comments refer from a first set of explorations conducted beginning of August, thus things likely will evolve.

→ To update the New Dash MVP performance Excel file, jump into the last section here.

Users in the MVP

This is your starting point. Mainly, what is available in the New Dash MVP from a Product Management POV. Figma link is here.

According to the screenshot, not all Users are in the MVP, but only a subset of the KYG Lite / hostfully / non airbnb/booking.com/agoda PMCs are available. This accounts for 22 users.

To retrieve these users, we can use:

select
	UserId as MvpUserId
from
	dbo.Claim
where
	ClaimType = 'KygMvp'
	and ClaimValue = 'true'

I am not sure what this Claim table is. Running a simple count on this exactly provides 22 users:

This can be joined easily with the already existing User table. I did a dedicated one-by-one check with the e-mails and I confirm these 22 users are the ones available in the User spreadsheet.

Moving on.

Basics of the MVP

Take a look again at the Figma for the MVP, this will clarify the following.

A Product Service is the detail of the Service being offered in New Dash. Not to be confused with a Product from the old dash.

**select** * **from** ProductService ps

Note: In some places is called Product, in other ProductService.

You will notice there’s both an Id and a ProductServiceFlag that has a binary (y/n) potential over 2 (2^0=1, 2^1=2, 2^2=4, etc). Keep this in mind.

A Product Bundle is a bundle of one or many different Product Services

**select** * **from** ProductBundle pb 

Currently in the MVP we only have 2 Product Bundles, namely Basic Screening and Basic Program. Each Product Bundle is associated to a Protection Plan with the ProtectionPlanId.

select * from ProtectionPlan pp 

At this stage, in the MVP, this table is not super informative since these Protection Plans are linked to just one different Protection each:

select * from Protection

In a nutshell, we have Basic Screening protection and IPRP Basic.

You will notice that we have a column called RequiredProductServices, that is either 1 or 257. My guess is that this informs of which Product Services are needed for a given Protection. Remember the ProductServiceFlag field? Well:

• Basic Screening ⇒ RequiredProductServices = 1 ⇒ ProductService.Id = 1 ⇒ Basic Screening
• IPRP Basic ⇒ RequiredProductServices = 257 = 1 + 256
  • ProductService.Id = 1 ⇒ Basic Screening
  • ProductService.Id = 256 ⇒ Waiver Pro

Meaning we have 2 Product Services (Basic Screening, Waiver Pro) currently available in 2 Product Bundles. Basic Screening service is available in both, and the only thing that changes is having or not Waiver Pro available within the Product Bundle.

From a business point of view, this would mean that:

The product bundle Basic Screening only contains the Product Service Basic Screening and it’s related to the Protection Basic Screening.

The product bundle Basic Program contains 2 product services. The first one, is the Basic Screening (same as before). The second one is the only paid service in the MVP and refers to Waiver Pro.

At this stage is worth to mention a couple of things:

• Basic screening it’s inherent for all users according to the MVP New Dash documentation. Likely it’s linked to DisplayOnFrontEnd field in the ProductBundle table, since I assume that if the user cannot see it, he/she cannot remove it. Needs confirmation.
• Basic screening it’s a free service, Waiver Pro is a payment service. More on this later 🙂

Moving on.

Pricing, billing, invoicing, protecting

This is not needed for what I am to do right now so it will need to be filled in later on, but I guess is important for the invoicing part of the requests.

select * from ProductServiceToPrice pstp 

In a nutshell, each ProductService (based on the ProductServiceId) has a different pricing (Amount) for each Currency (CurrencyId). Additionally, it can be Billed with different methods (BillingMethodId), Invoiced with different methods (InvoicedMethodId) and Paid with different types (PaymentTypeId).

Might be interesting to note that right now all UserProductBundleId are null, thus I assume at this stage no User has created a new Bundle other than the default.

A similar behaviour can be found for the Protections:

select * from ProtectionPlanToCurrency pptc 

In a nutshell, each Protection Plan (not Protection!) has different protection values (Baseline, Lower, Maximum) depending on the Currency (CurrencyId). In this case, I am not sure why we have SuperhogUserId here. Not clear to me the 3 different levels of protection.

This section can be improved with providing multi-joins with other tables. Tables be explored: BillingMethod, InvoicingMethod.

User has Product Bundles

Each User can have one or multiple Product Bundles.

Important: This relationship means that a User CAN apply a Product Bundle into a Listing, not that necessarily the Product Bundle IS USED.

In the case of no bundle applied, the default bundle applied is the Basic Screening. At this stage, all MVP users have the 2 Product Bundles: Basic Screening and Basic Program. The relationship User has Product Bundle is available in UserProductBundle.

Note

: there are MORE users in the UserProductBundle that the ones available in the MVP Users. This is because there was a release that went out on the 18th June (way before the MVP launch on 30th July, which coincides with the dates in which we start seeing Bookings with bundles beginning to be populated. It appears mainly Hostfully bookings are being populated as well as a few OwnerRez bookings. These two PMS are likely the only ones with webhooks enabled currently, so the assumption is that at this time when this release went out, we had already begun to attach product bundles to bookings in the back-end. This is also why non-MVP users were having product bundles created.

In the following queries I will just cap the data displayed for these 22 users.

with UserMVP AS (
select
	UserId as MvpUserId
from
	dbo.Claim
where
	ClaimType = 'KygMvp'
	and ClaimValue = 'true'
)
select 
	[User].CompanyName,
	UserProductBundle.*
from UserMVP
left join [User] 
on UserMVP.MvpUserId = [User].Id 
left join UserProductBundle  
on UserProductBundle.SuperhogUserId = UserMVP.MvpUserId

I intentionally hide SuperhogUserId, but you can see the visually how this table looks like by the CompanyName, available from User table. Total row count is 44 = 22x2. Some interesting subjects:

• We see how each user has 2 Product Bundles Id, from which we actually have the Name/Display Name in the table. Thus, no need to join UserProductBundle to ProductBundle (or at least, not at the moment). This looks a bit strange since usually the backend is fully normalised, but anyway.
• We also have here the ChoosenProductServices, with the values 1 and 257. So likely this is the way to retrieve “this bundle has these services”, as seen before. However I do not have the query to process it so at this stage I’m assuming so based on the business context.

A User assigns a Product Bundle to a Listing

Keep in mind that in the backend, Listing = Accommodation.

This information is available here:

select * from AccommodationToProductBundle atpb

It would make sense since we have the UserProductBundleId that is linked to an AccommodationId on a period of time StartDate - EndDate. However, as you can see, the table is empty. This means that no user has assigned a Product Bundle to a Listing yet.

A Booking comes from New Dash MVP

To see if a Booking comes from a New Dash MVP we should use BookingToProductBundle table.

select * from BookingToProductBundle btpb order by CreatedDate 

When exploring this table, you’ll see that there’s already information available from days created before the launch of the MVP (July 30th). This has been explained in this note here.

In order to filter by those that come from the New Dash MVP, we’ll force the join to retrieve Bookings that have UserProductBundle coming from Users that are in the MVP. In essence:

with UserMVP AS (
select
	UserId as MvpUserId
from
	dbo.Claim
where
	ClaimType = 'KygMvp'
	and ClaimValue = 'true'
)
select 
	[User].CompanyName,
	UserProductBundle.Name as ProductBundleName,
	BookingToProductBundle.*
from UserMVP
left join [User] 
on UserMVP.MvpUserId = [User].Id 
left join UserProductBundle  
on UserProductBundle.SuperhogUserId = UserMVP.MvpUserId
inner join BookingToProductBundle 
on BookingToProductBundle.UserProductBundleId = UserProductBundle.Id

However, we still see that for some of these MVP users, we have Bookings with Product Bundles created before the launch of the MVP. I’ll take an opinionated approach and enforce that a Bookings needs to have happened after the MVP launch on 30th of July; or at least, distinguish it in future queries.

Note: this is not perfect. If tomorrow we have a new user joining the MVP, we won’t be able to cut previous bookings at a dedicated migration date at user level. I checked and the Claim table does not contain any date related field, might be worth to dig deeper with Engineering team.

with UserMVP AS (
select
	UserId as MvpUserId
from
	dbo.Claim
where
	ClaimType = 'KygMvp'
	and ClaimValue = 'true'
)
select 
	[User].CompanyName,
	UserProductBundle.Name as ProductBundleName,
	BookingToProductBundle.*
from UserMVP
left join [User] 
on UserMVP.MvpUserId = [User].Id 
left join UserProductBundle  
on UserProductBundle.SuperhogUserId = UserMVP.MvpUserId
inner join BookingToProductBundle 
on BookingToProductBundle.UserProductBundleId = UserProductBundle.Id
where BookingToProductBundle.CreatedDate >= '2024-07-30'

So it seems we have 23 Bookings after the MVP launch, all with Basic Screening. This makes sense: since no Basic Program was applied into any listing it means that all of them should be coming from the Basic Screening bundle (the default) that only contains Basic Screening service.

From here I will take more opinionated considerations, for the sake of data quality. Since there’s a Start - End date, I assume this table can take Booking duplicates if there’s a change in the UserProductBundleId in time. My opinionated decision is taking the last updated row, mainly, the one that has EndDate as [NULL]. At the moment, all of them have these behavior so it does not matter that much.

At this stage, I assume that if at some point we have a Booking with a Product Bundle being Basic Program, it enforces the payment of the Waiver Pro service. I wonder if this will change in the future if a Product Bundle contains multiple Guest services (Waiver, Deposit): one thing is what the Guest sees, the other what he/she chooses. Since MVP is 0 guest interaction I assume it makes sense like this for a time, but might be worth to keep this in mind.

Tracking the MVP performance

Based on all these considerations, here’s a nice MVP query. Until we have this data in DWH with a proper PBI report, we need to survive by updating an Excel file.

Steps:

1. Run query below and export it to csv
2. Copy-paste the csv it into the Excel file attached below in the Raw Data tab (erasing any existing data)
3. On Excel, go to Data → Refresh All
4. Save with date suffix (_2024MMDD) and upload into Lou’s shared folder here.

Tracking query

This query could be optimised and improved. However I found it easier to understand how the data model looks like by taking additional steps. Just copy-paste this into DBeaver into Live prod schema and run.

/*
 
THIS QUERY RETRIEVES INFORMATION FOR THE MVP MINIMAL PERFORMANCE TRACKING
- Reads directly from the backend.
- Assumes MVP users come from Claim table with ClaimType as KygMvp and ClaimValue is true
- Assumes only retrieving the latest state, i.e., where EndDate is null
- Assumes paying services are all of those that are not in the ProductBundle Basic Screening
- For Bookings with Product Bundle, it excludes any booking before the MVP launch date on 30th July 2024

*/

with UserMVP AS (
select
	UserId as MvpUserId
from
	dbo.Claim
where
	-- THIS IS TO RETRIEVE MVP USERS --
	ClaimType = 'KygMvp'
	and ClaimValue = 'true'
), 
UserHasBundles AS (
select 
	UserMVP.MvpUserId,
	[User].CompanyName,
	COUNT(DISTINCT case when UserProductBundle.Name not in ('BasicScreening') then UserProductBundle.Id else null end) AS PaidService_UserProductBundleCount,
	COUNT(DISTINCT UserProductBundle.Id) AS UserProductBundleCount
from
	UserMVP
inner join [User] 
on
	UserMVP.MvpUserId = [User].Id
inner join UserProductBundle 
on
	UserMVP.MvpUserId = UserProductBundle.SuperhogUserId
where
	UserProductBundle.EndDate is null
group by 	
	UserMVP.MvpUserId,
	[User].CompanyName
),
BookingsPerUserBundle AS (
select 
	UserMVP.MvpUserId,
	UserProductBundle.Name as ProductBundleName,
	CAST(MIN(case when BookingToProductBundle.CreatedDate >= '2024-07-30' then BookingToProductBundle.CreatedDate else null end) as date) as FirstBookingWithBundleCreatedDate,
	CAST(MAX(case when BookingToProductBundle.CreatedDate >= '2024-07-30' then BookingToProductBundle.CreatedDate else null end) as date) as LastBookingWithBundleCreatedDate,
	COUNT(DISTINCT case when BookingToProductBundle.CreatedDate >= '2024-07-30' then BookingToProductBundle.BookingId else null end) as TotalBookingsWithBundle,
	COUNT(DISTINCT BookingToProductBundle.BookingId) as TotalBookingsWithBundle_FullHistory
from
	UserMVP
inner join UserProductBundle  
on
	UserProductBundle.SuperhogUserId = UserMVP.MvpUserId
inner join BookingToProductBundle 
on
	BookingToProductBundle.UserProductBundleId = UserProductBundle.Id
where
	BookingToProductBundle.EndDate is null
group by 
	UserMVP.MvpUserId,
	UserProductBundle.Name
),
ListingsPerUser AS (
select 
	UserMVP.MvpUserId,
	COUNT(DISTINCT Accommodation.AccommodationId) as TotalListings,
	COUNT(case when Accommodation.IsActive = 1 then Accommodation.AccommodationId else null end) as ActiveListings
from
	UserMVP
inner join AccommodationToUser
on
	UserMVP.MvpUserId = AccommodationToUser.SuperhogUserId
inner join Accommodation
on
	Accommodation.AccommodationId = AccommodationToUser.AccommodationId
group by 
	UserMVP.MvpUserId
), 
ListingsWithBundlePerUser AS (
select 
	UserMVP.MvpUserId,
	UserProductBundle.Name as ProductBundleName,
	CAST(MIN(AccommodationToProductBundle.CreatedDate) as date) as FirstListingWithBundleCreatedDate,
	CAST(MAX(AccommodationToProductBundle.CreatedDate) as date) as LastListingWithBundleCreatedDate,
	COUNT(DISTINCT AccommodationToProductBundle.AccommodationId) as TotalListingsWithBundle
from
	UserMVP
inner join UserProductBundle  
on
	UserProductBundle.SuperhogUserId = UserMVP.MvpUserId
inner join AccommodationToProductBundle
on
	AccommodationToProductBundle.UserProductBundleId = UserProductBundle.Id
where
	AccommodationToProductBundle.EndDate is null
group by 
	UserMVP.MvpUserId,
	UserProductBundle.Name
),
ListingAggregation AS (
select 
	ListingsPerUser.MvpUserId,
	AVG(ListingsPerUser.TotalListings) AS TotalListings,
	AVG(ListingsPerUser.ActiveListings) AS ActiveListings,
	MIN(case when ListingsWithBundlePerUser.ProductBundleName not in ('BasicScreening') then ListingsWithBundlePerUser.FirstListingWithBundleCreatedDate else null end) as PaidService_FirstListingWithBundleCreatedDate,
	MAX(case when ListingsWithBundlePerUser.ProductBundleName not in ('BasicScreening') then ListingsWithBundlePerUser.LastListingWithBundleCreatedDate else null end) as PaidService_LastListingWithBundleCreatedDate,
	SUM(case when ListingsWithBundlePerUser.ProductBundleName not in ('BasicScreening') then ListingsWithBundlePerUser.TotalListingsWithBundle else 0 end) as PaidService_TotalListingsWithBundle,
	MIN(ListingsWithBundlePerUser.FirstListingWithBundleCreatedDate) AS FirstListingWithBundleCreatedDate,
	MAX(ListingsWithBundlePerUser.LastListingWithBundleCreatedDate) AS LastListingWithBundleCreatedDate,
	SUM(ListingsWithBundlePerUser.TotalListingsWithBundle) AS TotalListingsWithBundle
from
	ListingsPerUser
left join ListingsWithBundlePerUser
on
	ListingsPerUser.MvpUserId = ListingsWithBundlePerUser.MvpUserId
group by
	ListingsPerUser.MvpUserId
),
BookingsAggregation AS (
select 
	BookingsPerUserBundle.MvpUserId,
	MIN(case when BookingsPerUserBundle.ProductBundleName not in ('BasicScreening') then BookingsPerUserBundle.FirstBookingWithBundleCreatedDate else null end) as PaidService_FirstBookingWithBundleCreatedDate,
	MAX(case when BookingsPerUserBundle.ProductBundleName not in ('BasicScreening') then BookingsPerUserBundle.LastBookingWithBundleCreatedDate else null end) as PaidService_LastBookingWithBundleCreatedDate,
	SUM(case when BookingsPerUserBundle.ProductBundleName not in ('BasicScreening') then BookingsPerUserBundle.TotalBookingsWithBundle else 0 end) as PaidService_TotalBookingsWithBundle,
	MIN(BookingsPerUserBundle.FirstBookingWithBundleCreatedDate) AS FirstBookingWithBundleCreatedDate,
	MAX(BookingsPerUserBundle.LastBookingWithBundleCreatedDate) AS LastBookingWithBundleCreatedDate,
	SUM(BookingsPerUserBundle.TotalBookingsWithBundle) AS TotalBookingsWithBundle
from
	BookingsPerUserBundle
group by
	BookingsPerUserBundle.MvpUserId
)
select 
	UserHasBundles.MvpUserId,
	UserHasBundles.CompanyName,
	UserHasBundles.UserProductBundleCount,
	UserHasBundles.PaidService_UserProductBundleCount,
	COALESCE(ListingAggregation.TotalListings,
	0) AS TotalListings,
	COALESCE(ListingAggregation.ActiveListings,
	0) AS ActiveListings,
	COALESCE(ListingAggregation.TotalListingsWithBundle,
	0) as TotalListingsWithBundle,
	COALESCE(ListingAggregation.PaidService_TotalListingsWithBundle,
	0) as PaidService_TotalListingsWithBundle,
	COALESCE(BookingsAggregation.TotalBookingsWithBundle,
	0) as TotalBookingsWithBundle,
	COALESCE(BookingsAggregation.PaidService_TotalBookingsWithBundle,
	0) as PaidService_TotalBookingsWithBundle,
	-- Listing Date Details --
	ListingAggregation.FirstListingWithBundleCreatedDate,
	ListingAggregation.LastListingWithBundleCreatedDate,
	ListingAggregation.PaidService_FirstListingWithBundleCreatedDate,
	ListingAggregation.PaidService_LastListingWithBundleCreatedDate,
	-- Booking Date Details --
	BookingsAggregation.FirstBookingWithBundleCreatedDate,
	BookingsAggregation.LastBookingWithBundleCreatedDate,
	BookingsAggregation.PaidService_FirstBookingWithBundleCreatedDate,
	BookingsAggregation.PaidService_LastBookingWithBundleCreatedDate
from
	UserHasBundles
left join ListingAggregation
on
	UserHasBundles.MvpUserId = ListingAggregation.MvpUserId
left join BookingsAggregation
on
	UserHasBundles.MvpUserId = BookingsAggregation.MvpUserId

Excel template

Example of Excel (use it as the template)

NewDashMVP_PerformanceTracking.xlsx

Example of export:

NewDashMVP_20240805.csv

Migrating this into the DWH

Pablo here. On my first day running this export (2024-08-12) I felt the frustration of being a human ETL machine and that pushed me to start cleaning this up already. So, I’m going to start to:

• Bring the tables from Core used in the above query into the DWH with Airbyte.
• Start modeling any intermediate entities that make sense.
• Make a reporting table with the same contents we’re doing here for the export.

Bring tables over

• The Core tables used in the export are:
  • dbo.Claim
  • dbo."User"
  • UserProductBundle
  • BookingToProductBundle
  • Accommodation
  • AccommodationToUser
  • AccommodationToProductBundle
• Staging models:
  • dbo.Claim
  • UserProductBundle
  • BookingToProductBundle
  • AccommodationToProductBundle
• Intermediate models:
  • int_core__user_product_bundle
  • int_core__accommodation_to_product_bundle
  • int_core__booking_to_product_bundle
  • int_core__new_dash_user_overview