1.4 KiB
1.4 KiB
- Intro and purpose
- We're here to show you Meltano
- With the idea of replacing Meltano and Airflow
- I want to give an overview of features and how it work
- The path forward is still not fully clear but this should give us an idea of whether we want to bother continuing this line of work
- Quick demo
- Assets and dependencies
- Materializing
- Asset checks
- Automations
- Extra
- Partitions
- Opentelemetry
- Run arbitrary pieces of code in task style, not asset style
- Architecture
- 4 components
- Responsibilities of each
- How I set it up locally
- Our repo and how this gets built
- Definitions
- Assets
- dlt replacing meltano
- Deployment
- Locally with docker compose, it's fast
- Single node or kubernetes
- Environment config and secrets
- Networking
- Pros and cons against current setup
- Pros
- Asset-centric is nicer, integration with dbt
- Sophisticated automation is posible
- Easy to model external dependencies
- More lightweight
- Logging and making sense of things is much friendlier
- No meltano for EL
- Cons
- Less popular tool
- No auth layer, must remain an internal system
- We need to do lift and shift while adjusting plumbing
- We need to rebuild (or extract) the file serving
- Less popular tool
- Pros