Case Study: A Daily Sales Pipeline

The design

Schedule 0 1 * * * (01:00 UTC) with catchup=False and max_active_runs=1. Tasks: a deferrable sensor waits for the day's export file; extract loads raw rows into staging.orders partitioned by {{ ds }}; a dbt build transforms them inside the warehouse into analytics.daily_orders using delete-then-insert for the date; a data-quality task checks row counts and null rates and fails the run if they are off; a final task refreshes the dashboard cache. Defaults: 3 retries with exponential backoff, a 1-hour execution_timeout, an on_failure_callback that alerts the team, and a source_db pool of 3 slots. DAG changes go through pull requests with an import test in CI. A backfill of any past week is safe because every task is idempotent and uses the run's date.

A dependable daily pipeline

Idempotent tasks, date-based partitions, retries, checks and alerts combine into a pipeline you can trust.

The pipeline on one page

Each line maps to a section of this course.

wait_for_export (deferrable sensor, timeout)  --> extract (partition dt={{ ds }})   (Sec 4, 3)
  --> dbt_build (delete+insert for the date, in warehouse)                         (Sec 5)
  --> check_rows (fails run on bad counts)  --> refresh_dashboard                  (Sec 6)
Defaults: retries=3 + backoff, execution_timeout=1h, on_failure_callback, pool source_db=3 (Sec 6)
Schedule: 0 1 * * *, catchup=False, max_active_runs=1                              (Sec 2)
Deploy: Git + PR review + DagBag import test, pinned versions, PostgreSQL metadata (Sec 7)

Write a runbook

For each alert, document what it means, how to check, how to re-run or backfill, and who owns it. A short runbook turns a 3 a.m. page into a routine fix.

Quick check: Why is a backfill of any past week safe in this design?

• Data is never stored
• Airflow forbids backfills
• Tasks are idempotent and use the run's date
• Because there are no retries