sh-notion/notion_team_organization/Private & Shared/Data Team Organisation 81ea09a1778c4ca2ab39e7f221730cb5/Data Team Organisation - Season 1 20f0446ff9c980d78a0bd614930586a3.md

Data Team Organisation - Season 1

🚨

This is an outdated document.

This document describes our ways of working between May ‘24, May ‘25. Early June ‘25 we reviewed how we worked and came up with an updated version of this. If you want to understand how we work, please refer to it. This page only exists for historical purposes.

Data Vision

🗣 *To foster a **data-driven culture** throughout the organization by providing easily accessible qualitative data, insightful analytics, and impactful data solutions to drive growth, innovation, and business value.*

Guiding Principles and Practices

📖 Transparency first

• We prioritize open and wide communication regarding current priorities and latest achievements. It is essential for everyone to understand whether we are working on a specific subject and, if not, why our efforts are focused elsewhere.
• To facilitate this, we will provide a clear and visual representation of task prioritization. This approach ensures that while not all tasks have fixed deadlines, the evolving nature of our backlog and the potential for new high-priority requests are acknowledged and managed effectively.
• Documentation can play a nice role in this area:
  • Data Catalogue: A comprehensive index of our data sources, data products, and reports, ensuring everyone knows what data is available and how to access it.
  • Data News: A weekly update detailing the latest developments and achievements of the Data team, keeping everyone informed of our progress and focus areas.
  • Data Papers: A collection of ad-hoc, in-depth analyses. These papers aim to share valuable insights and prevent the unnecessary repetition of similar analyses.

🏆 One team, one dream

• By collaborating: Encourage teamwork and cooperation across departments and teams. Foster an environment where everyone feels valued and their contributions are recognized.
• By sharing goals: Common objectives that everyone works towards, ensuring that success is measured by the collective achievements rather than individual ones.
• By granting mutual support: we promote a culture where team members support one another, share knowledge, and help each other overcome challenges. We encourage mentoring and peer-to-peer learning.

🧭 We are Analytical Ambassadors

• We actively encourage the use of data and analytics in everyday decision-making processes, as well as we promote the benefits of data-driven decisions through workshops, training sessions, and regular communication.
• We serve as evangelists by spreading awareness about the importance and advantages of using data across the organization.
• We offer support and guidance on best practices for data collection, analysis, interpretation, A/B tests, etc.

How do we collaborate with business teams?

Lines of Work

The Data team focuses on three primary lines of work, namely Maintenance (run), Projects (build), Ad-hoc Requests (business-oriented run).

1. Maintenance (run)

• Nature of work: This involves the ongoing tasks necessary to ensure that all data systems and processes are functioning correctly. These tasks are reactive by nature and often arise unexpectedly.
• Tracking: All maintenance work are tracked using a ticketing system in DevOps to keep a record of all work done.
• Time allocation: No constraint, since usually other products will depend on it. Ideally, it should be low and part of the build should aim to reduce Maintenance time.
• Estimation and priority: Given their unpredictable nature, these tasks are hard to estimate but are typically assigned top priority to minimize downtime and ensure seamless operation.
• Examples of tasks: Fixing data pipeline issues, resolving system outages, addressing data discrepancies, and ensuring the accuracy and availability of critical reports.

2. Projects (build)

• Nature of work: These involve long-term projects aimed at developing and enhancing mostly data products, but also data infrastructure, reporting capabilities, and workflows. These projects are strategic and contribute to the overall improvement of the data ecosystem.
• Tracking: High level initiatives will be made available at ProductBoard, to mimic Product team. These will be linked to Epics or Features in DevOps, that will be further divided and refined into DevOps user stories.
• Time allocation: No constraint. It should be the most important aspect after incident solving to ensure long-term growth capacity to the business.
• Estimation and priority: While initial estimates can be made, committing to exact timelines can be risky due to the complexity and potential scope changes. Projects are planned with flexibility to accommodate evolving requirements following an Agile iterative approach (bring value fast, iterate afterwards).
• Examples of tasks: Developing new data pipelines, creating comprehensive dashboards, implementing data governance frameworks, optimizing existing data processes, setting a new data quality alerting system, design and execution of A/B tests.

3. Ad-hoc Requests (business-oriented run)

• Nature of work: These are short-term, unplanned tasks that arise from immediate business needs or questions. The tasks are often small in scope and can be completed quickly.
• Tracking: All non-trivial ad-hoc requests are tracked using a ticketing system in DevOps to ensure they are addressed in a timely manner and to keep a record of all work done.
• Time allocation: Each week, a maximum of 10 hours is allocated for ad-hoc tasks. The responsibility for these tasks rotates weekly between team members (for the moment, Pablo and Uri), designated as the Data Captain.
• Estimation and priority: These requests, by their nature, might not need to be groomed with the rest of the team. It is the responsibility of the Data Captain to decide if this request can or needs to be handled right away or it can wait depending on the criticality. Common sense and business intuition should prevail.
• Examples of tasks: Generating quick reports, running specific data queries, and providing data insights for immediate business decisions.

Demand In-take Process

The process of taking in and managing requests from business teams is structured to ensure efficiency and clarity:

1. Request Submission:
   • The requests need to be submitted in the #data channel, via a Slack bot named Data Request. This should be the primary tool for business teams to submit their data requests. This ensures that all requests are captured in a centralized and accessible manner.
2. Triaging Requests:
   • The Data Captain is responsible for reviewing and categorizing incoming requests daily. Each request is triaged into one of the three lines of work:
     • Maintenance: Logged in the DevOps system to track ongoing support tasks.
     • Ad-hoc Requests: Also logged in DevOps, with a cap of 10 hours per week for the Data Captain to address these.
     • Build / Projects: Major requests are documented with a detailed business rationale, goals, and a high-level overview. These need to be discussed and refined. These are detailed on the product board. Afterwards, these will be broken down into actionable tasks with clear definitions of done, timelines, and business justifications into DevOps.

Communication and Priority Setting

Effective communication and clear priority setting are crucial for aligning the Data team’s efforts with business objectives:

1. Communication with main Project Stakeholders:
   • Tactical meetings are held with stakeholders involved in build projects. These meetings are focused on gather requirements, high level design, progress updates, addressing any issues, and aligning on project goals and timelines.
2. Priority Setting:
   • We assume mostly 3 sources of prioritisation:
     • For key, strategic decisions, quarterly high-level discussions are conducted with the TMT (Top Management Team).
     • For important decisions, more regular communication should happen with the Head of Product (Ben) and Finance Director (Suzannah).
     • For the rest of decisions, the Data team will be autonomous.
   • The first 2 types of discussions revolve around visualising the high-level product board, setting priorities based on business needs, and adjusting plans as necessary to ensure alignment with strategic objectives and ensuring a good team workload and capacity.

How do we collaborate internally within the Data team?

Agile:

• Go Kanban: The Official Guide to The Kanban Method | Kanban University
• Use AzureDevOps board. Main reason is to foster collaboration with tech teams if there’s dependencies with other Tech teams (we’ll be on the same tools)
• Dailies every morning to comment on updates (mostly following Kanban board)
• Team Retrospectives (once a month)
  • Both internal and with customers
• We do not do grooming sessions for the moment since it would be a bit overkill, we do on demand if needed for the time being

Communication:

• Internal communication through data-team-internal channel
• Data news: weekly basis summary towards everyone

Documentation:

• DWH + Reports/Data Products: → dbt model
• Notion
  • Data Catalog
  • Internal docs
  • Everything else
• Infra
  • git repo

Miscellaneous:

• Are we going to solve all requests? no. We’ll check first if it makes sense from a Data POV, considering the value of the actionable next step and the complexity of the request itself.

How do we collaborate with the tech team?

⛔ *This section is WIP since it´s less critical for the time being. Collaboration is ok so far*

• Data contracts
• PR notification
• Alignment on data models during design phases
• Regular communications:
  • Regular meetings Data x Tech (every 2 weeks?)
  • Dedicated channel to discuss tech subjects? Might be a bit overkill with the current org-size

Request handling

:WIP:

Incident Management

• We make our best effort to build postmortem reports to document and reflect on any incidents that take place in our systems. You can find our template here, and the list of incidents here.
• Any ongoing issues that are impacting business users or other dependants of the Data team should be announced in the #data slack channel. If the incident’s affect audience can be narrowed down to specific groups of users, please also try to notify them more directly to ensure they are aware of the situation.

Season 2 Proposal

In sh_invoicing_exporter, we