- Intro and purpose - We're here to show you Meltano - With the idea of replacing Meltano and Airflow - I want to give an overview of features and how it work - The path forward is still not fully clear but this should give us an idea of whether we want to bother continuing this line of work - Quick demo - Assets and dependencies - Materializing - Asset checks - Automations - Extra - Partitions - Opentelemetry - Run arbitrary pieces of code in task style, not asset style - Architecture - 4 components - Responsibilities of each - How I set it up locally - Our repo and how this gets built - Definitions - Assets - dlt replacing meltano - Deployment - Locally with docker compose, it's fast - - Single node or kubernetes - Environment config and secrets - Networking - Pros and cons against current setup - Pros - Asset-centric is nicer, integration with dbt - Sophisticated automation is posible - Easy to model external dependencies - More lightweight - Logging and making sense of things is much friendlier - No meltano for EL - Cons - Less popular tool - No auth layer, must remain an internal system - We need to do lift and shift while adjusting plumbing - We need to rebuild (or extract) the file serving - Less popular tool