Airflow DAG basics
A DAG defines tasks and dependencies so a workflow can run in the right order with retries and scheduling.
- Directed acyclic graph
- Tasks model units of work
- Scheduler triggers runs
Quick recall
374 cards
Batch vs streaming
Batch processes chunks on a schedule, while streaming processes events continuously with low latency.
- Batch is simpler
- Streaming lowers freshness delay
- Trade latency for complexity
ETL vs ELT
ETL transforms before loading, while ELT loads raw data first and transforms inside the warehouse later.
- ETL fits strict downstream schemas
- ELT fits scalable warehouses
- Tooling and cost model differ
Kafka basics
Kafka is a distributed log used for durable event streaming and decoupled producers and consumers.
- Topics store ordered partitions
- Consumers track offsets
- Great for event-driven pipelines
Orchestration vs transformation
Orchestration decides when and in what order work runs; transformation changes the data itself.
- Airflow schedules tasks
- dbt or Spark transform data
- Keep responsibilities clear
Parquet vs CSV vs JSON
CSV is simple but weakly typed, JSON is flexible but verbose, and Parquet is compressed columnar storage optimized for analytics.
- CSV is easy to inspect
- JSON handles nested structure
- Parquet is best for warehouse scans
Retries and dependencies in pipelines
Retries help recover transient failures, while dependencies prevent downstream work from running on incomplete upstream data.
- Retry only safe tasks
- Use backoff for transient issues
- Surface hard failures early
What are data quality checks?
Data quality checks validate assumptions like null rates, uniqueness, schema shape, and accepted value ranges.
- Catch bad data early
- Automate checks in pipelines
- Alert instead of silently passing errors
What are incremental loads?
Incremental loads process only new or changed data instead of reprocessing the full dataset every run.
- Use watermarks or timestamps
- Lower cost and faster runs
- Need late-arrival strategy
What does partitioning mean in data engineering?
Partitioning splits data into logical chunks so reads and writes can target less data and scale better.
- Common on date columns
- Improves pruning and performance
- Bad partition choices can hurt
What is a data lake?
A data lake stores large volumes of raw or semi-structured data cheaply for later processing.
- Raw and flexible storage
- Schema can be applied later
- Needs governance to avoid becoming messy
What is a data warehouse?
A data warehouse is a system optimized for analytical queries over integrated, historical business data.
- Read-heavy analytics store
- Often columnar
- Different goals from OLTP databases