43 lines
1.4 KiB
Markdown
43 lines
1.4 KiB
Markdown
|
|
||
|
|
- Intro and purpose
|
||
|
|
- We're here to show you Meltano
|
||
|
|
- With the idea of replacing Meltano and Airflow
|
||
|
|
- I want to give an overview of features and how it work
|
||
|
|
- The path forward is still not fully clear but this should give us an idea of whether we want to bother continuing this line of work
|
||
|
|
- Quick demo
|
||
|
|
- Assets and dependencies
|
||
|
|
- Materializing
|
||
|
|
- Asset checks
|
||
|
|
- Automations
|
||
|
|
- Extra
|
||
|
|
- Partitions
|
||
|
|
- Opentelemetry
|
||
|
|
- Run arbitrary pieces of code in task style, not asset style
|
||
|
|
- Architecture
|
||
|
|
- 4 components
|
||
|
|
- Responsibilities of each
|
||
|
|
- How I set it up locally
|
||
|
|
- Our repo and how this gets built
|
||
|
|
- Definitions
|
||
|
|
- Assets
|
||
|
|
- dlt replacing meltano
|
||
|
|
- Deployment
|
||
|
|
- Locally with docker compose, it's fast
|
||
|
|
-
|
||
|
|
- Single node or kubernetes
|
||
|
|
- Environment config and secrets
|
||
|
|
- Networking
|
||
|
|
- Pros and cons against current setup
|
||
|
|
- Pros
|
||
|
|
- Asset-centric is nicer, integration with dbt
|
||
|
|
- Sophisticated automation is posible
|
||
|
|
- Easy to model external dependencies
|
||
|
|
- More lightweight
|
||
|
|
- Logging and making sense of things is much friendlier
|
||
|
|
- No meltano for EL
|
||
|
|
- Cons
|
||
|
|
- Less popular tool
|
||
|
|
- No auth layer, must remain an internal system
|
||
|
|
- We need to do lift and shift while adjusting plumbing
|
||
|
|
- We need to rebuild (or extract) the file serving
|
||
|
|
- Less popular tool
|