From 2123b5237128d85166f63e85e410eee8dbe1c773 Mon Sep 17 00:00:00 2001 From: counterweight Date: Thu, 16 Jan 2025 23:48:56 +0100 Subject: [PATCH] lots of stuff --- ...want-code-defined-dashboards-so-badly.html | 123 ++++++++++++------ 1 file changed, 82 insertions(+), 41 deletions(-) diff --git a/public/writings/i-want-code-defined-dashboards-so-badly.html b/public/writings/i-want-code-defined-dashboards-so-badly.html index bfa02d6..e898c06 100644 --- a/public/writings/i-want-code-defined-dashboards-so-badly.html +++ b/public/writings/i-want-code-defined-dashboards-so-badly.html @@ -18,42 +18,53 @@

I want code defined dashboards so badly

-

For decades, Data teams and data gardeners in general have used specific tools to build dashboards. This - are also called reports, data tools, data products, and another gazillion funny names.

-

For the sake of clarity, when I say dashboard here, I refer to some piece of software that people look at - on a screen where they text, tables and charts, as well as a few controls to play with what they can see - (filters and selectors, mostly).

+

Analysts build dashboards. These are also called reports, data tools, data products, and another + gazillion funny names.

+ +

For the sake of clarity, when I say dashboard here, I refer to some software-driven wizardry that puts + pixels on a screen, displaying text, tables, and charts, as well as a few controls to play with what + data they can see (filters and selectors, mostly). Some human will read this in hope of understanding + data for something of use.

- Analysts and other species are expected to build lots of this. Some business person needs to know stuff, - so the analyst goes and builds a dashboard the person can look at. + Analysts and other species are expected to build lots of them. Some business person needs to know stuff, + so the analyst goes and builds a dashboard so the person can look at it and know the stuff he needs to.

- My description of a dashboard is very open, so there's a million ways to technically implement one. The - thing is, analysts are analyts, not software engineers. If I ask my good colleague Uri, who is a - wonderful analyst, to code you up your KPIs dashboard from scratch with React, he's going to take - somewhere between six weeks and two years. Not great. + My description of a dashboard is very open ended, technology wise. So there's a million ways to + technically implement one. The thing is, analysts are analysts, not software engineers. If I ask my good + colleague Uri, who is a wonderful analyst, to code you up your KPIs dashboard from scratch with React, + he's going to take somewhere between six weeks and two years. Not great. +

+

For decades, Data teams and data gardeners in general have used specific tools to build dashboards, which + are designed around the idea that an analyst is not a software engineer. This being the case, we've had + multiple generations of tools that have tried to make it simpler for people to build dashboards. The + whole point of them is it shouldn't take knowing linked lists and big O notation to declare "put what + comes out of this SELECT * FROM thingie into a pretty line chart".

- This being the case, we've had multiple generations of tools that have tried to make it simpler for - people to build dashboards. The whole point of them shouldn't take knowing linked lists and big O - notation to say "put what comes out of this `SELECT * FROM thingie` into a line chart". -

-

- These tools are abstractions, and like all abstractions, they will be opinionated, restrict your freedom - and be leaky to some degree. + These tools act as abstractions on the low-level details of rendering a screen with charts, and like all + abstractions, they will be opinionated, restrict your freedom and be leaky to some degree. In many cases, this is a great trade-off, + because the analyst (and his boss) really doesn't give a damn whether the stroke of the x-axis is 3px or + 5px + thick. But he does care about going from select to chart instantly.

At the time I'm writing this, I feel the most popular tools out there to do this work are Looker, - Tableau, Power BI. And all of them have something I hate deeply: You can't put a dashboard in a - text file, and you can't version control it with Git [1]. + Tableau, Power BI. All of them do a decent job in solving the above mentioned trade-off. Yet all three, + along with all the previous generations I've lived through, have something in common I hate deeply: + You can't store a dashboard in a text file, and you can't version control it with Git [1].

Instead, these tools will always ask you to build your dashboard through point and click, drag and drop - interfaces, and hide them behind propietary formats or just store it in their platform and not let you - see the raw thing behind. + interfaces, and then hide the underlying definition behind propietary formats or even worse, just store + it in their platform and not let you see the raw thing under the hood. You nasty nasty Google, you + gifted us mortals with LookML to build Explores and then threw us back in hell when time came to plot.

-

This causes an unfortunate set of limitations: +

This pattern of keeping the dashboard definitions kidnapped and away from us causes an unfortunate set of + limitations:

- - -

[1] You actually can put a Power BI dashboard in Git, but it's +

+ I'm hopeful this won't remain sci-fi for long. The world of data tooling has changed quite a bit in + the past decade, with a strong influence from software engineering and open source software. I remember + ten years ago the companies I was working with would bleed licenses for propietary databases, propietary + datawarehouses, propietary ETL tools, propietary governance tools and propietary visualization tools. + The idea that someone working in Business Intelligence (how this world was known before it turned into + simply "data") had to know how to use git seemed extraterrestrial. Now we have plenty of more modern + alternatives that allow a small company to easily run a full data stack in a simple VM with no licensing + costs. Composable, dev-friendly, open source, high quality tools have already conquered storage, + querying, transformations, ETL and governance. It's only a matter of time that the slot of the + visualization tool gets conquered as well. +

+

+ There are actually some extremely young tools which are already exploring ideas similar to this. evidence.dev is the best + one I've come accross so far. And even though it's young and still needs plenty of polishing and + feature-richness to be up there with the big boys that dominate the landscape, I think it's enough for + small teams as it is today. +

+

+ I hope that, some time soon, I can look back at this post and say my dream of code defined dashboards + has become a very mundane and boring reality, and that my junior analyts wherever I'm working at the + time think I'm a caveman when I tell them we used to update the name of a field across 231 Tableau + dashboards by opening, pointing and clicking for seveteen days when someone changed the field name in + the DWH.

+

[1] You actually can put a Power BI dashboard in Git, but it's quite useless since the best format they can offer you is an ocean of unreadable JSON-strings-within-JSONs you would never dare to touch without Power BI desktop,