- My tips and tricks when using Postgres as a DWH
- In November 2023, I joined Superhog (now called Truvi) to start out the Data team. As part of that, I
- also drafted and deployed the first version of its data platform.
-
- The context led me to choose Postgres for our DWH. In a time of Snowflakes, Bigqueries and Redshifts,
- this might surprise some. But I can confidently say Postgres has done a great job for us, and I can even
- dare to say it has provided a better experience than other, more trendy alternatives could have. I'll
- jot down my rationale for picking Postgres one of these days.
-
- Back to the topic: Postgres is not intended to act as a DWH, so using it as such might feel a bit hacky
- at times. There are multiple ways to make your life better with it, as well as related tools and
- practices that you might enjoy, which I'll try to list here.
-
- Use unlogged tables
- The Write Ahead Log comes active by default for the tables you create, and
- for good reasons. But in the context of an ELT DWH, it is probably a good idea to deactivate it by
- making your tables unlogged. Unlogged
- tables will provide you with much faster writes (roughly, twice as fast) which will make data
- loading and transformation jobs inside your DWH much faster.
-
- You pay a price for this with a few trade offs, the most notable being that if your Postgres server
- crashes, the contents of the unlogged tables will be lost. But,
- again, if you have an ELT DWH, you can survive by running a backfill. In Truvi, we made the decision to
- have the landing area for our DWH be logged, and everything else unlogged. This means if we experienced
- a crash (which still hasn't happened, btw), we would recover by running a full-refresh dbt run.
- If you are using dbt, you can easily apply this by adding this bit in your dbt_project.yml
- :
-
-models:
- +unlogged: true
-
-
- Tuning your server's parameters
- Postgres has many parameters you can fiddle with, with plenty of
- chances to either improve or destroy your server's performance.
- Postgres ships with some default values for it, which are almost surely not the optimal ones for
- your needs, specially if you are going to use it as a DWH. Simple changes like adjusting the
- work_mem will do wonders to speed up some of your heavier queries.
-
- There are many parameters to get familiar with and proper adjustment must be done taking your specific
- context and needs into account. If you have no clue at all, this little web app can give you some suggestions you
- canstart from.
-
- Running VACUUM ANALYZE right after building your tables
- Out of the box, Postgres will automatically run
- VACUUM
- and
- ANALYZE
- jobs automatically. The triggers that determine when each of those gets
- triggered can be adjusted with a few server parameters. If you follow an ELT pattern, most surely
- re-building your non-staging tables will cause Postgres to run them.
-
- But there's a detail that is easy to overlook. Postgres automatic triggers will start those quite fast,
- but not right after you build each table. This poses a performance issue: if your intermediate sections
- of the DWH have tables that build upon tables, rebuilding a table and then trying to rebuild a dependant
- without having an ANALYZE on the first one before might hurt you.
- Let me describe this with an example, because this one is a bit of a tongue twister: let's assume we have
- tables int_orders and int_order_kpis. int_orders holds all of our
- orders, and int_order_kpis derives some kpis from them. Naturally, first you will
- materialize int_orders from some upstream staging tables, and once that is complete, you
- will use its contents to build int_order_kpis.
-
-
- Having int_orders ANALYZE-d before you start building
- int_order_kpis is highly benefitial for your performance in building
- int_order_kpis. Why? Because having perfectly updated statistics and metadata on
- int_orders will help Postgres' query optimizer better plan the necessary query to
- materialize int_order_kpis. This can improve performance by orders of magnitude in some
- queries by allowing Postgres to pick the right kind of join strategy for the specific data you have, for
- example.
-
- Now, will Postgres auto VACUUM ANALYZE the freshly built int_orders before you
- start building int_order_kpis? Hard to tell. It depends on how you build your DWH, and how
- you've tuned your server's parameters. And the most dangerous bit is you're not in full control: it can
- be that sometimes it happens, and other times it doesn't. Flaky and annoying. Some day I'll
- write a post on how this behaviour drove me mad for two months because it made a model sometimes built
- in a few seconds, and other times in >20min.
-
-
- My advice is to make sure you always VACUUM ANALYZE right after building your tables. If
- you're using dbt, you can easily achieve this by adding this to your project's
- dbt_project.yml:
-
-models:
- +post-hook:
- sql: "VACUUM ANALYZE {{ this }}"
- transaction: false
- # ^ This makes dbt run a VACUUM ANALYZE on the models after building each.
- # It's pointless for views, but it doesn't matter because Postgres fails
- # silently withour raising an unhandled exception.
-
-
- Monitor queries with pg_stats_statements
- pg_stats_statements is an extension that nowadays ships with Postgres
- by default. If activated, it will log info on the queries executed in the server which you can check
- afterward. This includes many details, with how frequently does the query get called and what's the min,
- max and mean execution time being the ones you probably care about the most. Looking at those allows you
- to find queries that take long each time they run, and queries that get run a lot.
-
- Another important piece of info that gets recorded is who ran the query. This is helpful
- because, if you use users in a smart way, it can help you isolate expensive queries on different uses
- cases or areas. For example, if you use different users to build the DWH and to give your BI tool read
- access (you do that... right?), you can easily tell apart dashboard related queries from internal, DWH
- transformation ones. Another example could be internal reporting vs embedded analytics in your product:
- you might have stricter performance SLAs for product-embedded, customer-facing queries than for internal
- dashboards. Using different users and pg_stats_statements makes it possible for you to
- dissect performance issues on those separate areas independently.
- Dalibo's wonderful execution plan visualizer
- Sometimes you'll have some nasty query you just need to sit down with and optimize. In my experience, in
- a DWH this ends up happening with queries that involve many large tables in sequential joining and
- aggregation steps (as in, you join a few tables, group to some granularity, join some more, group again,
- etc).
-
- You can get the query's real execution details with EXPLAIN ANALYZE, but the output's
- readability is on par with morse-encoded regex patterns. I always had headaches dealing with them until
- I came across Dalibo's execution plan
- visualizer. You can paste the output of EXPLAIN ANALYZE there and see the query
- execution presented as a diagram. No amount of words will portray accurately how awesome the UX is, so
- I encourage you to try the tool with some nasty query and see for yourself.
- Local dev env + Foreign Data Wrapper
- One of the awesome things of using Postgres is how trivial it is to spin up an instance. This makes
- goofing around much more simpler than whenever setting up a new instance means paperwork, $$$, etc.
- Data team members at Truvi have a dockerized Postgres running in their laptops that they can use when
- they are developing on our DWH dbt project. In the early days, you could grab some production dump with
- some subset of tables from our staging layer and run significant chunks of our dbt DAG in your laptop if
- you were patient.
- A few hundreds of models later, this evolved to increasingly difficult and finally became impossible.
-
- Luckily, we came across Postgres' Foreign Data Wrapper. There's
- quite a bit to it, but to keep it short here, just be aware that FDW allows you to make a Postgres
- server give access to some table in a different Postgres server while pretending they are local. So, you
- query table X in Postgres server A, even though table X is actually stored in Postgres server B. But
- your query works just the same as if it was a local genuine table.
- Setting these up is fairly trivial, and has allowed our dbt project contributors to be able to execute
- hybrid dbt runs where some data and tables is local to their laptop, whereas some upstream data is being
- read from production server's. The approach has been great so far, enabling them to actually test models
- before commiting them to master in a convenient way.
-
- back to home
-
- 
- 
- 
- 
- 
- 
- 
-
-