Airflow DAG basics
A DAG defines tasks and dependencies so a workflow can run in the right order with retries and scheduling.
- Directed acyclic graph
- Tasks model units of work
- Scheduler triggers runs
Airflow DAG basics
Tagged with orchestration
3 cards
Orchestration vs transformation
Orchestration decides when and in what order work runs; transformation changes the data itself.
- Airflow schedules tasks
- dbt or Spark transform data
- Keep responsibilities clear
Orchestration vs transformation
Retries and dependencies in pipelines
Retries help recover transient failures, while dependencies prevent downstream work from running on incomplete upstream data.
- Retry only safe tasks
- Use backoff for transient issues
- Surface hard failures early
Retries and dependencies in pipelines