Batch vs streaming
Batch processes chunks on a schedule, while streaming processes events continuously with low latency.
- Batch is simpler
- Streaming lowers freshness delay
- Trade latency for complexity
Batch vs streaming
Tagged with etl
14 cards
ETL vs ELT
ETL transforms before loading, while ELT loads raw data first and transforms inside the warehouse later.
- ETL fits strict downstream schemas
- ELT fits scalable warehouses
- Tooling and cost model differ
ETL vs ELT
Orchestration vs transformation
Orchestration decides when and in what order work runs; transformation changes the data itself.
- Airflow schedules tasks
- dbt or Spark transform data
- Keep responsibilities clear
Orchestration vs transformation
Parquet vs CSV vs JSON
CSV is simple but weakly typed, JSON is flexible but verbose, and Parquet is compressed columnar storage optimized for analytics.
- CSV is easy to inspect
- JSON handles nested structure
- Parquet is best for warehouse scans
Parquet vs CSV vs JSON
Retries and dependencies in pipelines
Retries help recover transient failures, while dependencies prevent downstream work from running on incomplete upstream data.
- Retry only safe tasks
- Use backoff for transient issues
- Surface hard failures early
Retries and dependencies in pipelines
What are data quality checks?
Data quality checks validate assumptions like null rates, uniqueness, schema shape, and accepted value ranges.
- Catch bad data early
- Automate checks in pipelines
- Alert instead of silently passing errors
What are data quality checks?
What are incremental loads?
Incremental loads process only new or changed data instead of reprocessing the full dataset every run.
- Use watermarks or timestamps
- Lower cost and faster runs
- Need late-arrival strategy
What are incremental loads?
What does partitioning mean in data engineering?
Partitioning splits data into logical chunks so reads and writes can target less data and scale better.
- Common on date columns
- Improves pruning and performance
- Bad partition choices can hurt
What does partitioning mean in data engineering?
What is a data lake?
A data lake stores large volumes of raw or semi-structured data cheaply for later processing.
- Raw and flexible storage
- Schema can be applied later
- Needs governance to avoid becoming messy
What is a data lake?
What is a data warehouse?
A data warehouse is a system optimized for analytical queries over integrated, historical business data.
- Read-heavy analytics store
- Often columnar
- Different goals from OLTP databases
What is a data warehouse?