# Data Team Organisation

# Data Vision

<aside>
🗣 *To foster a **data-driven culture** throughout the organization by providing easily accessible qualitative data, insightful analytics, and impactful data solutions to drive growth, innovation, and business value.*

</aside>

# **Guiding Principles and Practices**

## **📖 Transparency first**

- We prioritize open and wide communication regarding current priorities and latest achievements. It is essential for everyone to understand whether we are working on a specific subject and, if not, why our efforts are focused elsewhere.
- To facilitate this, we will provide a clear and visual representation of task prioritization. This approach ensures that while not all tasks have fixed deadlines, the evolving nature of our backlog and the potential for new high-priority requests are acknowledged and managed effectively.
- Documentation can play a nice role in this area:
    - **Data Catalogue:** A comprehensive index of our data sources, data products, and reports, ensuring everyone knows what data is available and how to access it.
    - **Data News:** A weekly update detailing the latest developments and achievements of the Data team, keeping everyone informed of our progress and focus areas.
    - **Data Papers:** A collection of ad-hoc, in-depth analyses. These papers aim to share valuable insights and prevent the unnecessary repetition of similar analyses.

## **🏆 One team, one dream**

- By collaborating: Encourage teamwork and cooperation across departments and teams.
Foster an environment where everyone feels valued and their contributions are recognized.
- By sharing goals: Common objectives that everyone works towards, ensuring that success is measured by the collective achievements rather than individual ones.
- By granting mutual support: we promote a culture where team members support one another, share knowledge, and help each other overcome challenges. We encourage mentoring and peer-to-peer learning.

### **🧭 We are Analytical Ambassadors**

- We actively encourage the use of data and analytics in everyday decision-making processes, as well as we promote the benefits of data-driven decisions through workshops, training sessions, and regular communication.
- We serve as evangelists by spreading awareness about the importance and advantages of using data across the organization.
- We offer support and guidance on best practices for data collection, analysis, interpretation, A/B tests, etc.

# **How do we collaborate with business teams?**

## **Lines of Work**

The Data team focuses on three primary lines of work, namely Maintenance (run), Projects (build), Ad-hoc Requests (business-oriented run).

### **1. Maintenance (run)**

- **Nature of work:** This involves the ongoing tasks necessary to ensure that all data systems and processes are functioning correctly. These tasks are reactive by nature and often arise unexpectedly.
- **Tracking:** All maintenance work are tracked using a ticketing system in DevOps to keep a record of all work done.
- **Time allocation**: No constraint, since usually other products will depend on it. Ideally, it should be low and part of the build should aim to reduce Maintenance time.
- **Estimation and priority:** Given their unpredictable nature, these tasks are hard to estimate but are typically assigned top priority to minimize downtime and ensure seamless operation.
- **Examples of tasks:** Fixing data pipeline issues, resolving system outages, addressing data discrepancies, and ensuring the accuracy and availability of critical reports.

### **2. Projects (build)**

- **Nature of work:** These involve long-term projects aimed at developing and enhancing mostly data products, but also data infrastructure, reporting capabilities, and workflows. These projects are strategic and contribute to the overall improvement of the data ecosystem.
- **Tracking:** High level initiatives will be made available at ProductBoard, to mimic Product team. These will be linked to Epics or Features in DevOps, that will be further divided and refined into DevOps user stories.
- **Time allocation:** No constraint. It should be the most important aspect after incident solving to ensure long-term growth capacity to the business.
- **Estimation and priority:** While initial estimates can be made, committing to exact timelines can be risky due to the complexity and potential scope changes. Projects are planned with flexibility to accommodate evolving requirements following an Agile iterative approach (bring value fast, iterate afterwards).
- **Examples of tasks:** Developing new data pipelines, creating comprehensive dashboards, implementing data governance frameworks, optimizing existing data processes, setting a new data quality alerting system, design and execution of A/B tests.

### **3. Ad-hoc Requests (business-oriented run)**

- **Nature of work:** These are short-term, unplanned tasks that arise from immediate business needs or questions. The tasks are often small in scope and can be completed quickly.
- **Tracking:** All non-trivial ad-hoc requests are tracked using a ticketing system in DevOps to ensure they are addressed in a timely manner and to keep a record of all work done.
- **Time allocation:** Each week, **a maximum of 10 hours** is allocated for ad-hoc tasks. The responsibility for these tasks rotates weekly between team members (for the moment, Pablo and Uri), designated as the ***Data Captain***.
- **Estimation and priority:** These requests, by their nature, might not need to be groomed with the rest of the team. It is the responsibility of the Data Captain to decide if this request can or needs to be handled right away or it can wait depending on the criticality. Common sense and business intuition should prevail.
- **Examples of tasks:** Generating quick reports, running specific data queries, and providing data insights for immediate business decisions.

## **Demand In-take Process**

The process of taking in and managing requests from business teams is structured to ensure efficiency and clarity:

1. **Request Submission:**
    - The requests need to be submitted in the [***#data channel***](https://superhogteam.slack.com/archives/C06GFGHJD7H), via a Slack bot named ***Data Request.*** This should be the primary tool for business teams to submit their data requests. This ensures that all requests are captured in a centralized and accessible manner.
2. **Triaging Requests:**
    - The ***Data Captain*** is responsible for reviewing and categorizing incoming requests daily. Each request is triaged into one of the three lines of work:
        - **Maintenance:** Logged in the DevOps system to track ongoing support tasks.
        - **Ad-hoc Requests:** Also logged in DevOps, with a cap of 10 hours per week for the Data Captain to address these. They should be tagged with the `Data Captain` tag.
        - **Build / Projects:** Major requests are documented with a detailed business rationale, goals, and a high-level overview. These need to be discussed and refined. These are detailed on the product board. Afterwards, these will be broken down into actionable tasks with clear definitions of done, timelines, and business justifications into DevOps.

## **Communication and Priority Setting**

Effective communication and clear priority setting are crucial for aligning the Data team’s efforts with business objectives:

1. **Communication with main Project Stakeholders:**
    - Tactical meetings are held with stakeholders involved in build projects. These meetings are focused on gather requirements, high level design, progress updates, addressing any issues, and aligning on project goals and timelines.
2. **Priority Setting:**
    - We assume mostly 3 sources of prioritisation:
        - For key, strategic decisions, quarterly high-level discussions are conducted with the TMT (Top Management Team).
        - For important decisions, more regular communication should happen with the Matt Chetwood. We currently meet on a weekly basis.
        - For the rest of decisions, the Data team will be autonomous.
    - The first 2 types of discussions revolve around visualising the high-level product board, setting priorities based on business needs, and adjusting plans as necessary to ensure alignment with strategic objectives and ensuring a good team workload and capacity.

# How do we collaborate internally within the Data team?

Agile:

- Go Kanban: [The Official Guide to The Kanban Method | Kanban University](https://kanban.university/kanban-guide/)
- Use [AzureDevOps board](https://guardhog.visualstudio.com/Data/_boards/board/t/Data%20Team/Stories). Main reason is to foster collaboration with tech teams if there’s dependencies with other Tech teams (we’ll be on the same tools)
- Weeklies on Wednesdays
- Every day we all complete a written update. Read more below.
- Team Retrospectives (once a month)
- We do not do grooming sessions for the moment since it would be a bit overkill, we do on demand if needed for the time being

Communication:

- Internal communication through [data-team-internal](https://superhogteam.slack.com/archives/C072W6QB3UJ) channel
- [Data news](https://www.notion.so/Data-News-7dc6ee1465974e17b0898b41a353b461?pvs=21): weekly basis summary towards everyone
- Data Alerts for issues in our systems that need someone to look at.

Documentation:

- DWH + Reports/Data Products: → dbt model
- On our tools and infra: do check the Azure Devops Git repositories
- Anything else: here in Notion

## Usage of board

We have certain standards on how do we use [our Kanban task board](https://guardhog.visualstudio.com/Data/_boards/board/t/Data%20Team):

- Individual cards should be **obviously** doable in less than a week. If it isn’t, break it down into smaller chunks. (*Note that sometimes something will last more than a week because it is blocked by external factors. That’s fine).*
- Cards should provide in their description:
    - Context (why are we doing this? what’s the story behind it?).
    - As much detail as it makes sense on the task itself.
    - Note that, if you find yourself adding a wall of text, you might want to judge offloading the deep detail into Notion.
    - Related info (links to conversations, docs, deeper explanations, etc).
    - Accurate description on what needs to be achieved, including a definition of what does it mean for the task to be done in the Acceptance Criteria area.
    - As a reference, someone in the team should be able to pick the card and completely make sense out of it without asking you anything.
- Progress should be logged in the cards: update on a regular basis as you advance, encounter issues, further document, pivot because of new info, etc. How much? You judge. When in doubt, err on the side of over-logging.
    - As a reference, someone in the team should be able to pick the card while it’s WIP and continue where you left out without asking you anything.
    - Also, make sure the card is in the right lane and move it always as needed.
- When completing a card, please do leave a comment confirming that everything was done and add any info that might be relevant for future reference. The fact that the task was completed and what exactly was completed should be obvious to a reader.
- All cards should belong into the right epic. If it doesn’t have an epic, you either need to start an Epic or add it to the quarterly Unplanned epic. You decide what makes the most sense.
- Any Testing Data Alert that is not trivial should get a card. As a rule of thumb, if you have already invested already at least 5min in an alert and it’s still not solved, it is not trivial.

## Weekly meeting

We use our weekly meeting as our synchronous touchpoint to organize work and discuss whatever we need to.

The default agenda for this session is:

- Get a quick summary from each member on what was achieved since the last weekly.
- Discuss adhoc topics (whatever needs to be discussed in the team on a weekly basis)
- Coffee break after the previous, which shouldn’t take more than 30 min.
- Plan scopes for next week
    - We decide which tasks each of us will go for, committing for an amount of work that we feel is doable within a week
    - Note that it is expected that we all work on grooming the board continuously before this session so that we don’t need to define work to be done at this time.

You might wonder: what happens if I finish my weekly planned work before the next weekly? If this is the case (and hopefully it should be relatively often), feel free to continue working on whatever you think is most important. This is typically and ideal moment for those pesky refactors, cleanups, docs improvements, environment setups and other important but not urgent tasks that struggle to make their way into your agenda. 

## Daily Written Update

Every day, the team members write their Daily Written Update in the slack channel [#daily-written-update](https://teamtruvi.slack.com/archives/C0911ANQ7QS). In the update, we briefly answer the questions:

> *What did I do yesterday?*
> 
> 
> *What will I do today?*
> 
> *Is there anything blocking my tasks? If so, what is it?*
> 

It is expected to complete these everyday before 10:00. 

These updates allow the team to stay up to date with each other’s status without needing synchronous meetings. It also helps us compose our Data News and longer-term reflections on what we have accomplished.

## Data Captain duties

Each week, a member of the team holds the Data Captain role. The Data Captain acts as a first line of contact for our colleagues in other teams and is also tasked with some regular team tasks. Specifically, the Data Captain:

- Sends the Data News on Monday morning.
    - Note that the Captain does not write all the news. It is up to the team members to write their updates there.
- Triages any Data Testing alerts, documents them as a card in our board and sends them to their owner (which can be himself, of course).
- Handles any PBI app access requests.
- Sends the Business Targets weekly update to #all-staff on Fridays. [See this one by Uri as an example](https://teamtruvi.slack.com/archives/C6CKB771Q/p1749194573318649).

# Incident Management

- We make our best effort to build postmortem reports to document and reflect on any [incidents that take place in our systems](https://www.notion.so/Incident-Management-4829884213d744d4884be6c53988e696?pvs=21). You can find [our template here](https://www.notion.so/20241104-01-Booking-invoicing-incident-due-to-bulk-UpdatedDate-change-82f0fde01b83440e8b2d2bd6839d7c77?pvs=21), and [the list of incidents here](https://www.notion.so/Incident-Reports-9cdecb44c3914d24a0075ca1e8958fbf?pvs=21).
- Any ongoing issues that are impacting business users or other dependants of the Data team should be announced in the `#data` slack channel. If the incident’s affect audience can be narrowed down to specific groups of users, please also try to notify them more directly to ensure they are aware of the situation.

Subpages

[Data Team Organisation - Season 1](Data%20Team%20Organisation%2081ea09a1778c4ca2ab39e7f221730cb5/Data%20Team%20Organisation%20-%20Season%201%2020f0446ff9c980d78a0bd614930586a3.md)

[Season 2 Proposal](Data%20Team%20Organisation%2081ea09a1778c4ca2ab39e7f221730cb5/Season%202%20Proposal%2020e0446ff9c9805ca36bfa696e9e319c.md)