Executors and Deployment Options

Where tasks actually run

The executor decides how tasks are run. The LocalExecutor runs them as processes on the scheduler machine (simple, fine for small setups). The CeleryExecutor distributes tasks to a pool of long-running worker machines through a message broker (Redis or RabbitMQ): scalable and fast to start. The KubernetesExecutor launches a fresh pod per task: strong isolation and per-task resources, with startup overhead. Recent Airflow versions can also run multiple executors together. You can run Airflow yourself (Docker Compose, the official Helm chart on Kubernetes) or use a managed service (Amazon MWAA, Google Cloud Composer, Astronomer) that handles upgrades and scaling for a fee.

From one machine to a team platform

Choose an executor, ship DAGs through Git and CI, and test them before they reach the scheduler.

Choosing an executor

A starting guide; benchmark your own workload.

Learning / tiny team          LocalExecutor (or standalone)
Steady load, many short tasks   CeleryExecutor (Redis/RabbitMQ + worker pool)
Mixed/heavy tasks, isolation    KubernetesExecutor (pod per task) or KubernetesPodOperator
Small platform team, want less ops   managed service (MWAA / Composer / Astronomer)

Use PostgreSQL for the metadata DB

SQLite is only for learning. Production deployments use PostgreSQL (or MySQL) for the metadata database, with backups and monitoring, because the scheduler and every task read and write it.

Quick check: Which executor launches a new pod for each task?

• No executor does this
• LocalExecutor
• CeleryExecutor
• KubernetesExecutor