Backfills and Reprocessing

Replay history on purpose

A backfill runs a DAG for past data intervals, for example after fixing a bug or adding a new column. Because your tasks are idempotent and use the run's dates, backfilling is just running the same logic for many dates. Start the backfill with the Airflow CLI or UI for a specific date range, limit concurrency (max_active_runs) so it does not overwhelm your sources, test on a small range first, and watch downstream consumers: reprocessed data may trigger alerts or refresh dashboards.

Backfill from the CLI

Syntax depends on the Airflow version, so check airflow dags backfill --help (Airflow 2) or the Airflow 3 backfill command/UI. Test one day first.

# Airflow 2.x style (illustrative)
airflow dags backfill orders_etl \
  --start-date 2026-09-01 --end-date 2026-09-07 \
  --reset-dagruns

# limit parallelism in the DAG definition: max_active_runs=2

Backfill small, then scale

A backfill that rewrites a year of data in one go can overload databases and downstream dashboards. Do one day, verify, then widen.

Quick check: What makes backfills safe?

• Appending without keys
• Using datetime.now()
• Idempotent tasks that use the run's dates
• Running everything at once with no limits