For decades, Data teams and data gardeners in general have used specific tools to build dashboards. This - are also called reports, data tools, data products, and another gazillion funny names.
-For the sake of clarity, when I say dashboard here, I refer to some piece of software that people look at - on a screen where they text, tables and charts, as well as a few controls to play with what they can see - (filters and selectors, mostly).
+Analysts build dashboards. These are also called reports, data tools, data products, and another + gazillion funny names.
+ +For the sake of clarity, when I say dashboard here, I refer to some software-driven wizardry that puts + pixels on a screen, displaying text, tables, and charts, as well as a few controls to play with what + data they can see (filters and selectors, mostly). Some human will read this in hope of understanding + data for something of use.
- Analysts and other species are expected to build lots of this. Some business person needs to know stuff, - so the analyst goes and builds a dashboard the person can look at. + Analysts and other species are expected to build lots of them. Some business person needs to know stuff, + so the analyst goes and builds a dashboard so the person can look at it and know the stuff he needs to.
- My description of a dashboard is very open, so there's a million ways to technically implement one. The - thing is, analysts are analyts, not software engineers. If I ask my good colleague Uri, who is a - wonderful analyst, to code you up your KPIs dashboard from scratch with React, he's going to take - somewhere between six weeks and two years. Not great. + My description of a dashboard is very open ended, technology wise. So there's a million ways to + technically implement one. The thing is, analysts are analysts, not software engineers. If I ask my good + colleague Uri, who is a wonderful analyst, to code you up your KPIs dashboard from scratch with React, + he's going to take somewhere between six weeks and two years. Not great. +
+For decades, Data teams and data gardeners in general have used specific tools to build dashboards, which
+ are designed around the idea that an analyst is not a software engineer. This being the case, we've had
+ multiple generations of tools that have tried to make it simpler for people to build dashboards. The
+ whole point of them is it shouldn't take knowing linked lists and big O notation to declare "put what
+ comes out of this SELECT * FROM thingie into a pretty line chart".
- This being the case, we've had multiple generations of tools that have tried to make it simpler for - people to build dashboards. The whole point of them shouldn't take knowing linked lists and big O - notation to say "put what comes out of this `SELECT * FROM thingie` into a line chart". -
-- These tools are abstractions, and like all abstractions, they will be opinionated, restrict your freedom - and be leaky to some degree. + These tools act as abstractions on the low-level details of rendering a screen with charts, and like all + abstractions, they will be opinionated, restrict your freedom and be leaky to some degree. In many cases, this is a great trade-off, + because the analyst (and his boss) really doesn't give a damn whether the stroke of the x-axis is 3px or + 5px + thick. But he does care about going from select to chart instantly.
At the time I'm writing this, I feel the most popular tools out there to do this work are Looker, - Tableau, Power BI. And all of them have something I hate deeply: You can't put a dashboard in a - text file, and you can't version control it with Git [1]. + Tableau, Power BI. All of them do a decent job in solving the above mentioned trade-off. Yet all three, + along with all the previous generations I've lived through, have something in common I hate deeply: + You can't store a dashboard in a text file, and you can't version control it with Git [1].
Instead, these tools will always ask you to build your dashboard through point and click, drag and drop - interfaces, and hide them behind propietary formats or just store it in their platform and not let you - see the raw thing behind. + interfaces, and then hide the underlying definition behind propietary formats or even worse, just store + it in their platform and not let you see the raw thing under the hood. You nasty nasty Google, you + gifted us mortals with LookML to build Explores and then threw us back in hell when time came to plot.
-This causes an unfortunate set of limitations: +
This pattern of keeping the dashboard definitions kidnapped and away from us causes an unfortunate set of + limitations:
Which brings me to my craving which heads this post: I want code defined dashboards so badly.
+Which brings me to my craving which heads this post: I want code defined dashboards so + badly.
Picture a tool (I'm going to call my dream tool) where you can define a dashboard in a plain text file - with some formal structure. The file - defines what data gets pulled in (queries), how it gets displayed (visuals and formatting), and some - other additional elements (filters and controls, text for descriptions, headers, etc.). Possibly, the - syntax also allows for metadata and comments, allowing the analyst to document the dashboard in the same - file, instead of in some external tool that will eventually drift away from reality because I'm lazy and - can't have two windows open at the same time. My dream tool is capable of reading this file and making a - dashboard out of it.
+ with some formal structure. The file defines what data gets pulled in (queries), how it gets displayed + (visuals and formatting), and some other additional elements (filters and controls, text for + descriptions, headers, etc.). Possibly, the syntax also allows for metadata and comments, allowing the + analyst to document the dashboard in the same file, instead of in some external tool that will + eventually drift away from reality because I'm lazy and can't have two windows open at the same time. My + dream tool is capable of reading this file, figuring out how to get the needed data, and rendering a + dashboard out of it. Notice I still don't care whether the stroke of the x-axis is 3px or 5px thick. I + don't need the full power (and responsibility) of web development. Just that the whole deal is stored in + a plain text file. +Okay, so let's assume my dream tool exists and works like that. What have we gained? A lot of things!
@@ -102,23 +117,49 @@fulfilled_orders into
+ completed_orders in the latest MySQL migration and forgot to update the dashboard?
+ Not to worry, our CI tests caught that before it hit master.
[1] You actually can put a Power BI dashboard in Git, but it's
+
+ I'm hopeful this won't remain sci-fi for long. The world of data tooling has changed quite a bit in
+ the past decade, with a strong influence from software engineering and open source software. I remember
+ ten years ago the companies I was working with would bleed licenses for propietary databases, propietary
+ datawarehouses, propietary ETL tools, propietary governance tools and propietary visualization tools.
+ The idea that someone working in Business Intelligence (how this world was known before it turned into
+ simply "data") had to know how to use git seemed extraterrestrial. Now we have plenty of more modern
+ alternatives that allow a small company to easily run a full data stack in a simple VM with no licensing
+ costs. Composable, dev-friendly, open source, high quality tools have already conquered storage,
+ querying, transformations, ETL and governance. It's only a matter of time that the slot of the
+ visualization tool gets conquered as well.
+
+ There are actually some extremely young tools which are already exploring ideas similar to this. evidence.dev is the best
+ one I've come accross so far. And even though it's young and still needs plenty of polishing and
+ feature-richness to be up there with the big boys that dominate the landscape, I think it's enough for
+ small teams as it is today.
+
+ I hope that, some time soon, I can look back at this post and say my dream of code defined dashboards
+ has become a very mundane and boring reality, and that my junior analyts wherever I'm working at the
+ time think I'm a caveman when I tell them we used to update the name of a field across 231 Tableau
+ dashboards by opening, pointing and clicking for seveteen days when someone changed the field name in
+ the DWH. [1] You actually can put a Power BI dashboard in Git, but it's
quite useless since the best format they
can offer you is an ocean of unreadable JSON-strings-within-JSONs you would never dare to touch
without Power BI desktop,