Airflow DAG basics
A DAG defines tasks and dependencies so a workflow can run in the right order with retries and scheduling.
- Directed acyclic graph
- Tasks model units of work
- Scheduler triggers runs
Airflow DAG basics
ETL / Data Engineering cards
16 cards
Batch vs streaming
Batch processes chunks on a schedule, while streaming processes events continuously with low latency.
- Batch is simpler
- Streaming lowers freshness delay
- Trade latency for complexity
Batch vs streaming
ETL vs ELT
ETL transforms before loading, while ELT loads raw data first and transforms inside the warehouse later.
- ETL fits strict downstream schemas
- ELT fits scalable warehouses
- Tooling and cost model differ
ETL vs ELT
Kafka basics
Kafka is a distributed log used for durable event streaming and decoupled producers and consumers.
- Topics store ordered partitions
- Consumers track offsets
- Great for event-driven pipelines
Kafka basics
Orchestration vs transformation
Orchestration decides when and in what order work runs; transformation changes the data itself.
- Airflow schedules tasks
- dbt or Spark transform data
- Keep responsibilities clear
Orchestration vs transformation
Parquet vs CSV vs JSON
CSV is simple but weakly typed, JSON is flexible but verbose, and Parquet is compressed columnar storage optimized for analytics.
- CSV is easy to inspect
- JSON handles nested structure
- Parquet is best for warehouse scans
Parquet vs CSV vs JSON
Retries and dependencies in pipelines
Retries help recover transient failures, while dependencies prevent downstream work from running on incomplete upstream data.
- Retry only safe tasks
- Use backoff for transient issues
- Surface hard failures early
Retries and dependencies in pipelines
What are data quality checks?
Data quality checks validate assumptions like null rates, uniqueness, schema shape, and accepted value ranges.
- Catch bad data early
- Automate checks in pipelines
- Alert instead of silently passing errors
What are data quality checks?
What are incremental loads?
Incremental loads process only new or changed data instead of reprocessing the full dataset every run.
- Use watermarks or timestamps
- Lower cost and faster runs
- Need late-arrival strategy
What are incremental loads?
What does partitioning mean in data engineering?
Partitioning splits data into logical chunks so reads and writes can target less data and scale better.
- Common on date columns
- Improves pruning and performance
- Bad partition choices can hurt
What does partitioning mean in data engineering?