Merged PR 5675: Challenge for DE

New open challenges focused more on each domain

Related work items: #29825

README.md

@@ -11,4 +11,7 @@ Make a folder for your challenge and store all related documents and artifacts t
 
 ## Inventory
 
+- `excel_leveling_test`: An Excel-based assessment designed to evaluate the current skill level of participants, covering key areas such as formulas, merging data and pivot tables.
 - `basic-excel-challenge`: a basic excel challenge that tests some skills that would be expected for someone to have basic analyst role.
+- `basic_sql_training`: A set of SQL learning materials and exercises designed to build foundational SQL skills and familiarity with the DWH.
+- `data_engineering_challenge`: An EL+transformation case to judge the skills of a Data Engineer. Designed to be both a takehome assignment and an interview discussion tool.

challenges/basic_sql_training/README.md

@@ -0,0 +1,46 @@
+# Basic SQL Training
+
+A training folder to help participants get familiar with SQL and our Data Warehouse (DWH). It includes a mix of learning material, beginner-level exercises with answers, and more open-ended challenges based on real internal requests.
+
+This training is intended for Domain Analysts or anyone looking to improve their ability to query and explore data independently. Early exercises focus on foundational SQL and navigating the DWH, while later challenges mimic real-life data requests and require more analytical thinking.
+
+## Learning Material
+- Complete [SQLBolt.com](https://sqlbolt.com/) chapters 1-12 + Subqueries
+- Fiddle around `O'Reilly Head First SQL.pdf` Chapters 1, 6 and 8
+
+## Goals
+
+The training encourages the development of the following skills:
+
+- Understand and explore DWH tables and schemas
+- Write basic SQL queries using `SELECT`, `WHERE`, `GROUP BY`, `ORDER BY`
+- Perform joins between tables using primary and foreign keys
+- Use aggregation functions like `COUNT`, `SUM`, `AVG`, `MIN`, `MAX`
+- Apply filtering logic and subqueries
+- Export and analyze results in Excel or Power BI
+- Present findings clearly and concisely
+
+The early exercises have defined answers to help build confidence, while the open-ended challenges simulate real data tickets and require the use of domain knowledge, assumptions, and creativity.
+
+## Inventory
+
+- `O'Reilly Head First SQL.pdf`: This thoroughly revised book teaches you SQL fundamentals and how to really take advantage of it.
+- `SQL Training Exercises.doc`: Basic SQL questions with provided solutions to get familiar with the syntax and the DWH structure. In DOCX format for editing.
+- `SQL Training Exercises.pdf`: Same content as the DOC version, in PDF format for easy sharing with participants.
+- `SQL Open Challenges.doc`: Advanced SQL challenges based on real internal data requests ("Data Captain tickets"). These are open-ended and don't include answers.
+- `SQL Open Challenges.pdf`: Same content as the DOC version, in PDF format for easy sharing with participants.
+
+
+## How to Use
+
+- Share the learning material with the participants.
+- Have some learning sessions for the participants to get more familiarized with our DWH.
+- Provide the exercises and challenges once they have finished studying the basics of SQL.
+- Set a timeline for them and review after finishing.
+
+## Solutions
+
+The basic exercises come with the solutions included.
+The Open Challenges do not.
+
+

challenges/data_engineering_challenge/README.md

@@ -0,0 +1,30 @@
+# Data Engineering Challenge
+
+## Description
+
+This challenge has been designed for judging candidates. It can be used both as a takehome exercise that allows for async filtering as well as for a case interview.
+
+It provides the opportunity to judge:
+
+- Tooling
+  - Git
+  - Docker
+  - Some SQL database
+  - Python
+- Skills
+  - The ability of the candidate to communicate and document his work.
+  - Systems thinking and architecture design.
+  - Software Development practices (testing, code quality, etc).
+  - Data Engineering practices (concerns on scale, maintainability, flexibility of solution, etc).
+
+## How to use
+
+- First, send the contents of `candidate_deliverables`. I advise giving no strict time limit, just whatever is practical for the constraints of the selection process.
+- Once the candidate submits his solution, judge it.
+  - If the solution comes with instructions and/or automations that allow running it, try it.
+  - You can get some inspiration on how to judge from `takehome_assessment_guidelines.md`.
+  - There is no real strict threshold for what is "good enough". My advice is to (1) discard any candidate that doesn't manage to provide at least a barebones, working solution and (2) for the remaining candidates, follow through with the interview and rank them depending on quality of delivery + discussion.
+- If you follow through, the next step I recommend is an interview to discuss the solution in detail. You can use the `interview_guidelines.md` to get some ideas on how to handle such interview and how to judge the candidate.
+- After judging both the takehome and the interview performance, make a decision.
+
+If you find that the set up is "too demanding" and is scaring candidates away, you can lower your standards by pivoting towards no takehome part. Instead, only use the case as a theoretical situation for the case interview. I would personally send the description the day before the interview so they can go through it.

challenges/data_engineering_challenge/candidate_deliverables/challenge.md

@@ -0,0 +1,44 @@
+# Truvi Data Engineering Challenge
+
+Welcome to Truvi's Data Engineering Challenge. This challenge is a small, toy sample of the kind of work we do on a daily basis. Please read through the challenge in detail. Reach out to the recruiting manager if anything is unclear.
+
+## Context
+
+Truvi has recently signed a contract with House Group Holiday Rentals (HG), a Property Management Company. HG operates holiday properties for multiple Owner Companies. This means the properties are strictly owned by each the Owner Companies, but HG takes care of running the rental operation for them. As part of their services, they have hired Truvi to protect the bookings that happen in the properties of the Owner Companies. Our specific agreement is that:
+
+- All bookings that happen in their properties will be protected by Truvi (that is, if the guest does any damage to the property, we will cover for it).
+- In exchange, Truvi will invoice 10GBP/14USD/12EUR (depending on the country of the Owner Company) per booking on check-out date.
+- Invoicing will be done in monthly cycles. That means that each Owner Company will get one invoice, each month, with the bookings that checked out within the month.
+- Besides, there's a minimum fee per Owner Company and month of 100GBP/140USD/120EUR. If there are no bookings, or the fees for the bookings that happen in a month don't add up to those amounts, the minimum fee will still be charged to the Owner Company.
+
+To implement the invoicing of this deal, we will need to integrate with the systems of HG, fetch their booking data and process it.
+
+## What we need
+
+The goal of the challenge is to build a toy architecture that integrates with our customer's system, ingests their data and processes it to deliver a clean summary table for our Data Analysts and finance team colleagues. The final table must display, for each Owner Company and month, what is their revenue, in both their original currency as well as converted into GBP.
+
+## Challenge and constraints
+
+We would like you to set up a small system to ingest and process the data described above. The goal is to end up with a running SQL database that holds the table described in the previous section (we will simply refer to it as "the final table").
+
+You have been delivered a folder named `fake_api`. You can check the `README.md` in that folder to run a toy fake HTTP API that mocks the customer's system. We expect you to use it. You are not expected to modify, iterate or improve this API in any way.
+
+You should also have received a CSV named `currency_rates.csv` with currency exchange rates.
+
+Your solution should:
+
+- Create a SQL database. Feel free to use whatever SQL database you feel comfortable with.
+- Build some way to ingest the data held in the fake API into the database. Feel free to build it as similar as to what you deliver in a production system (NOTE: we understand this is a hiring test and you have limited time and energy. We DON'T expect a PERFECT, PRODUCTION grade delivery... but the more quality you pack, the better we will appreciate your skills. It's also OK to consciously not make some bits perfect and then proactively discuss in the interview how would you build such parts in a real environment so we can learn about your ideas).
+- Load the data in `currency_rates.csv` into the database to use it as part of your transformations. You can do this in any way you want, dirty and unsustainable even. We won't judge this bit other than the data being loaded. For the case, we will just assume that the real context would provide you with timely currency rates data.
+- Within the database, do some transformations to deliver the final table.
+- Include a way to easily print part of the final table contents.
+
+Some guidelines:
+
+- Please, deliver your solution via a Github repo that contains all relevant code, files, docs and other artifacts.
+- We usually work on Linux systems, so we would appreciate if your solution is runnable on Linux, Mac or WSL.
+- We would appreciate if your example is clear and has enough documentation so that someone can run it just by reading your submission. You can assume this person knows their way around whatever tooling you're using. You can safely assume that we're happy running this in our laptops, there's no need to bother with any sophisticated infra.
+- Feel free to tackle the deployment of the SQL database in whatever way you want, but we feel using a Docker container is the simplest approach for our context. Again, you can do it differently, just keep in mind we would like to be able to execute your solution.
+- Feel free to ignore anything related to authentication and security. Even if we would care about such topics in production, we won't bother for this challenge.
+- Even though orchestration and monitoring are important topics that we will surely discuss with you in the interview, we're not expecting your solution to address those. It's fine if your solution gets only run once and has little output other than some terminal output.
+- Feel free to add any additional documentation, explainers, human-readable bits you find relevant. If you need to make assumptions when building your solution, we encourage you to list them, for example.

challenges/data_engineering_challenge/candidate_deliverables/fake_api/README.md

@@ -0,0 +1,111 @@
+# Fake API
+
+A tiny script + CSV to fake the API of House Group Holiday Rentals.
+
+## How to run
+
+- Create a dedicated `venv` and install the packages listed in `requirements.txt`.
+- Move your terminal to this directory and run `python3 fake_api.py`.
+- This will start the API and serve it on port `5000` until you stop it with Ctrl+C.
+
+## How to request
+
+The API has a single endpoint `/api/bookings`. This endpoint implements pagination, sorting and some basic filtering. You can find below a few `curl` calls and their responses to get an idea of how the API works. You can start a new terminal after you got the API running and use them to test that the API is running fine.
+
+- Bookings for a certain check-in date:
+
+```bash
+curl "http://localhost:5000/api/bookings?check_in_date=2024-10-01"
+
+{
+  "page": 1,
+  "per_page": 2,
+  "results": [
+    {
+      "booking_id": "b3c810d6-8ab9-41f5-9810-ed809d1d1c64",
+      "check_in_date": "2024-06-20 22:12:04",
+      "check_out_date": "2024-06-24 22:12:04",
+      "owner_company": "Garcia, Hamilton and Carr",
+      "owner_company_country": "USA"
+    },
+    {
+      "booking_id": "e019d228-3fff-46dd-b54e-a98364a5399a",
+      "check_in_date": "2024-12-02 01:16:23",
+      "check_out_date": "2024-12-05 01:16:23",
+      "owner_company": "Campos PLC",
+      "owner_company_country": "France"
+    }
+  ],
+  "total": 1000
+}
+```
+
+- Bookings for a country, with pagination being used:
+
+```bash
+curl "http://localhost:5000/api/bookings?owner_company_country=France&page=1&per_page=3"
+
+{
+  "page": 1,
+  "per_page": 3,
+  "results": [
+    {
+      "booking_id": "e019d228-3fff-46dd-b54e-a98364a5399a",
+      "check_in_date": "2024-12-02 01:16:23",
+      "check_out_date": "2024-12-05 01:16:23",
+      "owner_company": "Campos PLC",
+      "owner_company_country": "France"
+    },
+    {
+      "booking_id": "b0cd14f7-5bdd-4cc1-900c-4f193b26c0ae",
+      "check_in_date": "2024-04-11 03:57:51",
+      "check_out_date": "2024-04-21 03:57:51",
+      "owner_company": "Campos PLC",
+      "owner_company_country": "France"
+    },
+    {
+      "booking_id": "ca678537-d032-4f4e-9135-9fbba287b00d",
+      "check_in_date": "2024-11-08 15:24:11",
+      "check_out_date": "2024-11-11 15:24:11",
+      "owner_company": "Campos PLC",
+      "owner_company_country": "France"
+    }
+  ],
+  "total": 97
+}
+```
+
+- Sorted and paginated:
+
+```bash
+curl "http://localhost:5000/api/bookings?sort_by=check_out_date&sort_order=desc&page=1&per_page=3"
+
+{
+  "page": 1,
+  "per_page": 3,
+  "results": [
+    {
+      "booking_id": "007d0909-1dc2-4a0d-bb3f-a925321bd09b",
+      "check_in_date": "2024-12-29 22:39:18",
+      "check_out_date": "2024-12-31 22:39:18",
+      "owner_company": "Campos PLC",
+      "owner_company_country": "France"
+    },
+    {
+      "booking_id": "bffccb7b-f3a8-40bc-9142-4f6964a3e44a",
+      "check_in_date": "2024-12-29 21:11:07",
+      "check_out_date": "2024-12-31 21:11:07",
+      "owner_company": "Faulkner-Howard",
+      "owner_company_country": "UK"
+    },
+    {
+      "booking_id": "f71fefb6-38ef-4b9a-b5a6-08014417b91f",
+      "check_in_date": "2024-12-28 20:12:36",
+      "check_out_date": "2024-12-31 20:12:36",
+      "owner_company": "Jones, Jefferson and Rivera",
+      "owner_company_country": "USA"
+    }
+  ],
+  "total": 1000
+}
+```

challenges/data_engineering_challenge/candidate_deliverables/fake_api/fake_api.py

@@ -0,0 +1,60 @@
+from flask import Flask, jsonify, request
+import csv
+
+app = Flask(__name__)
+CSV_FILE = 'fake_bookings.csv'
+
+ALLOWED_FIELDS = [
+    "booking_id",
+    "check_in_date",
+    "check_out_date",
+    "owner_company",
+    "owner_company_country"
+]
+
+def load_data():
+    with open(CSV_FILE, newline='') as csvfile:
+        return list(csv.DictReader(csvfile))
+
+# Load data once at startup
+bookings = load_data()
+
+@app.route('/api/bookings')
+def get_bookings():
+    filters = request.args
+    filtered = bookings.copy()
+
+    # --- Filtering ---
+    for key, value in filters.items():
+        if key in ALLOWED_FIELDS:
+            filtered = [item for item in filtered if value.lower() in item.get(key, '').lower()]
+
+    # --- Sorting ---
+    sort_by = filters.get("sort_by")
+    sort_order = filters.get("sort_order", "asc").lower()
+    if sort_by in ALLOWED_FIELDS:
+        filtered.sort(
+            key=lambda x: x.get(sort_by, "").lower(),
+            reverse=(sort_order == "desc")
+        )
+
+    # --- Pagination ---
+    try:
+        page = int(filters.get("page", 1))
+        per_page = int(filters.get("per_page", 10))
+    except ValueError:
+        return jsonify({"error": "Invalid pagination values"}), 400
+
+    start = (page - 1) * per_page
+    end = start + per_page
+    paginated = filtered[start:end]
+
+    return jsonify({
+        "total": len(filtered),
+        "page": page,
+        "per_page": per_page,
+        "results": paginated
+    })
+
+if __name__ == '__main__':
+    app.run(debug=True)

challenges/data_engineering_challenge/candidate_deliverables/fake_api/requirements.txt

@@ -0,0 +1 @@
+flask

challenges/data_engineering_challenge/interview_guidelines.md

@@ -0,0 +1,76 @@
+# Interview Guidelines
+
+This document proposes how to execute the interview AFTER the submission of a solution to the challenge.
+
+## Structure
+
+- Begin by allowing the candidate to provide an overview of the solution he has implemented. This should be high level, not in full detail. 5min ideally, in no case longer than 10min.
+- From there, you can take the discussion wherever you feel like. I suggest three different tactics, which you can distribute across your slotted time however you see fit:
+  - Discuss why he has done things the way he has.
+  - Ask the candidate about how he would do things better/differently in production.
+  - Propose changes or additional requirements and discuss with the candidate how we would modify his solution to deal with them.
+
+I'll drop a mix of such questions by skill topic below. Asking all of them in an interview is impossible time-wise, so just take it as inspiration, not a checklist. I personally never know what I'll ask exactly, since the most interesting things to ask always depend on what the candidate has built.
+
+### About the requirements
+
+- Why...
+  - Did you make the assumptions you made? (Basically, discuss on the assumptions the candidate did).
+- In production...
+  - What additional information would you ask for?
+  - How would you manage the relationship with the technical team in the customer company? What would you expect from them?
+  - How would you document the delivery for the customers of your work? How could we provide access to the data to them?
+- Imagine...
+  - That bookings can have their checkin and checkout dates changed. Would that affect your approach? What would need to change?
+
+### About the database choice
+
+- Why...
+  - Have you picked this specific database?
+- In production...
+  - Would you pick the same database? Another one? Why?
+- Imagine...
+  - We would need to use Postgres specifically for this challenge. Would there be any limitations/feature lackness? What would change in your solution? Would you use something Postgres has and your chosen database doesn't?
+
+### On performance
+
+- In production...
+  - At what data volume do you think your solution would stop working? Where would the bottleneck/issue sit on? What would you do?
+  - How would your design change if the data delivered by the customer API was immutable or was not immutable? (Events vs Updateable booking entity)
+- Imagine...
+  - We suddenly need extremely low latency in processing. As soon as a booking is filled in the customer systems, we need to adjust to it. Perhaps we must visibly show the "ongoing" bill to the customer company. How could we achieve this near real-time processing?
+
+### On Testing & Quality
+
+- In production...
+  - How would you test ingestion pipelines? What kinds of failures would you write tests for?
+  - What unit vs integration tests would you prioritize?
+- Imagine...
+  - You discovered after deployment that the customer API occasionally returns duplicate or inconsistent booking records. How would you test and protect against that?
+
+### On extensibility
+
+- In production...
+  - Would you feel comfortable with the extensibility of the current code or not? Why?
+- Imagine...
+  - We have three more customers like House Group Holiday Rentals that we want to integrate. Agreement, business rules, data needs are the same. They all have similar data around bookings, but they have different structures and IT systems in place. What parts of our current solution are general to all of them, which are specific to House Group Holiday Rentals? What would we need to change to onboard the new customers?
+
+### On monitoring
+
+- In production...
+  - How would we monitor this whole thing? What would you look for? What tooling would you like to use?
+  - How could we organize the team to deal with tackling whatever could go wrong? Actually... what are all the things that can go wrong here?
+- Imagine...
+  - The customer systems are flaky and sometimes fail and have outages. What would you change in your solution?
+
+### On security
+
+- In production...
+  - Where would security risks be present? How would we secure things down appropiately?
+
+### On infrastructure
+
+- In production...
+  - What kind of infra would you set up to deploy this?
+- Imagine...
+  - You are starting from Truvi's current infra and architecture (this would require you to discuss our architecture with the candidate so he's familiar with it). How would you go about building and deploying this in our current one?

challenges/data_engineering_challenge/takehome_assessment_guidelines.md

@@ -0,0 +1,41 @@
+# Takehome assessment guidelines
+
+Fundamental, basic stuff:
+
+- Did the candidate deliver with a complete, self-contained git repo?
+- Is their solution sufficiently documented to understand what is going on?
+- Can you run it? Is it a smooth process, or do you have to figure out lots of stuff along the way? Note that some degree of figuring things out and tweaking on our side is expected given that the candidate doesn't know what platform his solution will be running on.
+
+Regarding the business context:
+
+- Did he understand all the requirements properly, or is there some obvious confusion?
+- Regarding delivering:
+  - Did he attribute bookings to the check out date?
+  - Did he use the right currency for each country?
+  - Did he implement the minimum fee correctly?
+  - Did he try to tackle somehow combinations of Owner Company and month that have 0 bookings?
+
+Regarding the usage of git:
+
+- Did he do small, purposeful commits? Or did he blast a few or just one monster commit?
+- Note that I wouldn't just discard someone if they make a single monster commit, but if they display step by step, small increments behaviour, that's good.
+
+Regarding the deployment:
+
+- Is it complete and exhaustive? Does he clearly list dependencies?
+- Does he provide you with easy tools and utilities to run things?
+- Did he include any way to test his deployment?
+
+Regarding the Extract and Load from the fake API to the database:
+
+- Is it clean and readable?
+- Did he put any measures in place for logging?
+- Does his solution load data in one single batch, or in smaller batches?
+- Does he do
+
+Regarding SQL:
+
+- Is it clean and readable?
+- Did he bother with stuff like indices, constraints, primary keys, etc?
+- Did he do intermediate, modular tables/views to get to the final table? Or did he just roll one huge query?
+- Did he use any fancier features from his particular database choice that display mastery of it? For example, using a PG materialized view + a trigger to update the final table any time a new record is ingested from the fake API.

challenges/excel_leveling_test/README.md

@@ -0,0 +1,36 @@
+# Excel Leveling Test
+
+An Excel leveling tests to test your current leven and skills with Excel. A data-team grade data analyst should fly through this. A Domain Analyst is expected to be able to complete it in the proposed timeline, with possible minor mistakes.
+
+## Goals
+
+The challenge requires the candidate to put to use the following skills:
+
+- Be able to read existing tables and understand what's in them.
+- Formatting data to make it more readable.
+- Perform filters on a clean table.
+- Run basic arithmetic formulas.
+- Perform calculations and adding new data.
+- Import data from a csv file.
+- Merge tables based on more than 1 field.
+- Create pivot tables.
+- Creating graphs and draw conclusions from it.
+
+Some of these can be solved manually, through formulas, with pivot tables, etc. It's possible that different candidates use different strategies.
+
+Besides this, you can emphasize more or less the importance of pretty result presentation, reproducibility, methodology explanations, etc. That's up to you and not really defined in the test.
+
+## Inventory
+
+- `Excel_Level_Test.xlsx`: the workbook that serves as both starting point and instructions for the challenge. Candidate can use a copy of it as his working area.
+- `currency_data.csv`: a supplementary file that the candidate will have to import into the workboox.
+
+## How to use
+
+- Provide the candidate with both `Excel_Level_Test.xlsx` and `currency_data.csv`.
+- Sprinkle any specific instructions you might want to add.
+- Send it all to the candidate, provide a timeline and sit tight.
+
+## Solutions
+
+We will go through them together with the candidates.