stuf

dagster-demo.md

@@ -0,0 +1,43 @@
+
+- Intro and purpose
+    - We're here to show you Meltano
+    - With the idea of replacing Meltano and Airflow
+    - I want to give an overview of features and how it work
+    - The path forward is still not fully clear but this should give us an idea of whether we want to bother continuing this line of work
+- Quick demo
+    - Assets and dependencies
+    - Materializing
+    - Asset checks
+    - Automations
+    - Extra
+        - Partitions
+        - Opentelemetry
+        - Run arbitrary pieces of code in task style, not asset style
+- Architecture
+    - 4 components
+    - Responsibilities of each
+    - How I set it up locally
+- Our repo and how this gets built
+    - Definitions
+    - Assets
+    - dlt replacing meltano
+- Deployment
+    - Locally with docker compose, it's fast
+    - 
+    - Single node or kubernetes 
+    - Environment config and secrets
+    - Networking
+- Pros and cons against current setup
+    - Pros
+        - Asset-centric is nicer, integration with dbt
+        - Sophisticated automation is posible
+        - Easy to model external dependencies
+        - More lightweight
+        - Logging and making sense of things is much friendlier
+        - No meltano for EL
+    - Cons
+        - Less popular tool
+        - No auth layer, must remain an internal system
+        - We need to do lift and shift while adjusting plumbing
+        - We need to rebuild (or extract) the file serving 
+        - Less popular tool