Apache Airflow: Orchestrate Data Pipelines

Learn to schedule and monitor workflows with Apache Airflow: DAGs, TaskFlow, scheduling, XComs, branching, sensors, retries, idempotent design and deployment.

Start course →

Syllabus

Airflow Basics

1. What Airflow Is and Is Not
2. Architecture: Scheduler, Executor, Workers, Database
3. Installing and Running Locally
4. Core Concepts: DAG, Task, Run, Instance

Writing DAGs

1. Operators and Providers
2. The TaskFlow API
3. Setting Dependencies
4. Scheduling: Cron, Intervals and Catchup

Passing Data and Templating

1. XComs and Their Limits
2. Jinja Templating and Macros
3. Connections, Variables and Secrets

Control Flow

1. Branching
2. Trigger Rules
3. Sensors and Deferrable Operators
4. Dynamic Task Mapping

Designing Reliable Pipelines

1. Idempotent, Partitioned Tasks
2. Backfills and Reprocessing
3. Keep Airflow Lightweight

Reliability and Operations

1. Retries, Timeouts and Callbacks
2. Concurrency, Pools and Priorities
3. Monitoring, Logs and SLAs

Scaling and Deployment

1. Executors and Deployment Options
2. Deploying DAGs with Git and CI

Putting It Together

1. Case Study: A Daily Sales Pipeline
2. Revision: Cheat Sheet and Self-Check