+ Hi, Pablo here +
+ ++
My tips and tricks when using Postgres as a DWH
+In November 2023, I joined Superhog (now called Truvi) to start out the Data team. As part of that, I + also drafted and deployed the first version of its data platform. +
+The context led me to choose Postgres for our DWH. In a time of Snowflakes, Bigqueries and Redshifts, + this might surprise some. But I can confidently say Postgres has done a great job for us, and I can even + dare to say it has provided a better experience than other, more trendy alternatives could have. I'll + jot down my rationale for picking Postgres one of these days.
++ Back to the topic: Postgres is not intended to act as a DWH, so using it as such might feel a bit hacky + at times. There are multiple ways to make your life better with it, as well as related tools and + practices that you might enjoy, which I'll try to list here. +
+Use unlogged tables
+ The Write Ahead Log comes active by default for the tables you create, and
+ for good reasons. But in the context of an ELT DWH, it is probably a good idea to deactivate it by
+ making your tables unlogged. Unlogged
+ tables will provide you with much faster writes (roughly, twice as fast) which will make data
+ loading and transformation jobs inside your DWH much faster.
+
You pay a price for this with a few trade offs, the most notable being that if your Postgres server + crashes, the contents of the unlogged tables will be lost. But, + again, if you have an ELT DWH, you can survive by running a backfill. In Truvi, we made the decision to + have the landing area for our DWH be logged, and everything else unlogged. This means if we experienced + a crash (which still hasn't happened, btw), we would recover by running a full-refresh dbt run.
+If you are using dbt, you can easily apply this by adding this bit in your dbt_project.yml
+ :
+models:
+ +unlogged: true
+ Tuning your server's parameters
+Postgres has many parameters you can fiddle with, with plenty of + chances to either improve or destroy your server's performance.
+Postgres ships with some default values for it, which are almost surely not the optimal ones for
+ your needs, specially if you are going to use it as a DWH. Simple changes like adjusting the
+ work_mem will do wonders to speed up some of your heavier queries.
+
There are many parameters to get familiar with and proper adjustment must be done taking your specific + context and needs into account. If you have no clue at all, this little web app can give you some suggestions you + canstart from. +
+Running VACUUM ANALYZE right after building your tables
+ Out of the box, Postgres will automatically run
+ VACUUM
+ and
+ ANALYZE
+ jobs automatically. The triggers that determine when each of those gets
+ triggered can be adjusted with a few server parameters. If you follow an ELT pattern, most surely
+ re-building your non-staging tables will cause Postgres to run them.
+
But there's a detail that is easy to overlook. Postgres automatic triggers will start those quite fast,
+ but not right after you build each table. This poses a performance issue: if your intermediate sections
+ of the DWH have tables that build upon tables, rebuilding a table and then trying to rebuild a dependant
+ without having an ANALYZE on the first one before might hurt you.
Let me describe this with an example, because this one is a bit of a tongue twister: let's assume we have
+ tables int_orders and int_order_kpis. int_orders holds all of our
+ orders, and int_order_kpis derives some kpis from them. Naturally, first you will
+ materialize int_orders from some upstream staging tables, and once that is complete, you
+ will use its contents to build int_order_kpis.
+
+ Having int_orders ANALYZE-d before you start building
+ int_order_kpis is highly benefitial for your performance in building
+ int_order_kpis. Why? Because having perfectly updated statistics and metadata on
+ int_orders will help Postgres' query optimizer better plan the necessary query to
+ materialize int_order_kpis. This can improve performance by orders of magnitude in some
+ queries by allowing Postgres to pick the right kind of join strategy for the specific data you have, for
+ example.
+
Now, will Postgres auto VACUUM ANALYZE the freshly built int_orders before you
+ start building int_order_kpis? Hard to tell. It depends on how you build your DWH, and how
+ you've tuned your server's parameters. And the most dangerous bit is you're not in full control: it can
+ be that sometimes it happens, and other times it doesn't. Flaky and annoying. Some day I'll
+ write a post on how this behaviour drove me mad for two months because it made a model sometimes built
+ in a few seconds, and other times in >20min.
+
+ My advice is to make sure you always VACUUM ANALYZE right after building your tables. If
+ you're using dbt, you can easily achieve this by adding this to your project's
+ dbt_project.yml:
+
+models:
+ +post-hook:
+ sql: "VACUUM ANALYZE {{ this }}"
+ transaction: false
+ # ^ This makes dbt run a VACUUM ANALYZE on the models after building each.
+ # It's pointless for views, but it doesn't matter because Postgres fails
+ # silently withour raising an unhandled exception.
+ Monitor queries with pg_stats_statements
+ pg_stats_statements is an extension that nowadays ships with Postgres + by default. If activated, it will log info on the queries executed in the server which you can check + afterward. This includes many details, with how frequently does the query get called and what's the min, + max and mean execution time being the ones you probably care about the most. Looking at those allows you + to find queries that take long each time they run, and queries that get run a lot. +
+Another important piece of info that gets recorded is who ran the query. This is helpful
+ because, if you use users in a smart way, it can help you isolate expensive queries on different uses
+ cases or areas. For example, if you use different users to build the DWH and to give your BI tool read
+ access (you do that... right?), you can easily tell apart dashboard related queries from internal, DWH
+ transformation ones. Another example could be internal reporting vs embedded analytics in your product:
+ you might have stricter performance SLAs for product-embedded, customer-facing queries than for internal
+ dashboards. Using different users and pg_stats_statements makes it possible for you to
+ dissect performance issues on those separate areas independently.
Dalibo's wonderful execution plan visualizer
+Sometimes you'll have some nasty query you just need to sit down with and optimize. In my experience, in + a DWH this ends up happening with queries that involve many large tables in sequential joining and + aggregation steps (as in, you join a few tables, group to some granularity, join some more, group again, + etc). +
+You can get the query's real execution details with EXPLAIN ANALYZE, but the output's
+ readability is on par with morse-encoded regex patterns. I always had headaches dealing with them until
+ I came across Dalibo's execution plan
+ visualizer. You can paste the output of EXPLAIN ANALYZE there and see the query
+ execution presented as a diagram. No amount of words will portray accurately how awesome the UX is, so
+ I encourage you to try the tool with some nasty query and see for yourself.
Local dev env + Foreign Data Wrapper
+One of the awesome things of using Postgres is how trivial it is to spin up an instance. This makes + goofing around much more simpler than whenever setting up a new instance means paperwork, $$$, etc.
+Data team members at Truvi have a dockerized Postgres running in their laptops that they can use when + they are developing on our DWH dbt project. In the early days, you could grab some production dump with + some subset of tables from our staging layer and run significant chunks of our dbt DAG in your laptop if + you were patient.
+A few hundreds of models later, this evolved to increasingly difficult and finally became impossible. +
+Luckily, we came across Postgres' Foreign Data Wrapper. There's + quite a bit to it, but to keep it short here, just be aware that FDW allows you to make a Postgres + server give access to some table in a different Postgres server while pretending they are local. So, you + query table X in Postgres server A, even though table X is actually stored in Postgres server B. But + your query works just the same as if it was a local genuine table.
+Setting these up is fairly trivial, and has allowed our dbt project contributors to be able to execute + hybrid dbt runs where some data and tables is local to their laptop, whereas some upstream data is being + read from production server's. The approach has been great so far, enabling them to actually test models + before commiting them to master in a convenient way.
++ +