Lesson 1 / 25

What Airflow Is and Is Not

Describe Airflow as a workflow orchestrator and know what it should not be used for.

An orchestrator, not a data engine

Apache Airflow is an open-source platform to author, schedule and monitor workflows. A workflow is a DAG (directed acyclic graph) of tasks with dependencies, defined in ordinary Python. Airflow decides when things run, in what order, retries failures and records the history in a UI. It is an orchestrator: it should tell Spark, a warehouse, dbt or an API to do the heavy work, not process large datasets inside its own workers. It is built for batch workflows with a clear schedule, not for real-time streaming.

Define in Python, run on a schedule

You write workflows as Python code; Airflow schedules them, runs the tasks in order and shows you what happened.

Four stages: define, schedule, execute, observe.
Figure 1.1 — Define, schedule, execute and observe.

A railway control room

The control room sets timetables, signals which train may proceed and records delays. It does not pull the trains itself; the engines do. Airflow sets and monitors the timetable; your systems do the pulling.

Pick the right tool

For event streams use Kafka or Flink; for a single cron job on one server a plain cron entry may be enough. Choose Airflow when you have many dependent steps, need retries, history and visibility.

Quick check: What is Airflow best at?

  • Real-time stream processing
  • Processing huge datasets inside its own workers
  • Scheduling and monitoring batch workflows of dependent tasks
  • Serving web pages
Answer

Scheduling and monitoring batch workflows of dependent tasks — Airflow orchestrates; the heavy lifting should happen in systems built for it.