# Case Study: A Daily Sales Pipeline — Apache Airflow: Orchestrate Data Pipelines

Source: https://www.geekswithgeeks.com/en/airflow/wrap-case

> Design a daily pipeline that loads orders, builds a revenue table and refreshes a dashboard.

## The design

Schedule `0 1 * * *` (01:00 UTC) with `catchup=False` and `max_active_runs=1`. Tasks: a **deferrable sensor** waits for the day's export file; **extract** loads raw rows into `staging.orders` partitioned by `{{ ds }}`; a **dbt build** transforms them inside the warehouse into `analytics.daily_orders` using **delete-then-insert** for the date; a **data-quality** task checks row counts and null rates and fails the run if they are off; a final task refreshes the dashboard cache. Defaults: 3 retries with exponential backoff, a 1-hour `execution_timeout`, an `on_failure_callback` that alerts the team, and a `source_db` pool of 3 slots. DAG changes go through pull requests with an import test in CI. A backfill of any past week is safe because every task is idempotent and uses the run's date.

## A dependable daily pipeline

Idempotent tasks, date-based partitions, retries, checks and alerts combine into a pipeline you can trust.

![Four parts: extract, transform, check, publish.](assets/figures/airflow/section-8-map.svg) — Figure 8.1 — Extract, transform, check and publish.

## The pipeline on one page

Each line maps to a section of this course.

```text
wait_for_export (deferrable sensor, timeout)  --> extract (partition dt={{ ds }})   (Sec 4, 3)
  --> dbt_build (delete+insert for the date, in warehouse)                         (Sec 5)
  --> check_rows (fails run on bad counts)  --> refresh_dashboard                  (Sec 6)
Defaults: retries=3 + backoff, execution_timeout=1h, on_failure_callback, pool source_db=3 (Sec 6)
Schedule: 0 1 * * *, catchup=False, max_active_runs=1                              (Sec 2)
Deploy: Git + PR review + DagBag import test, pinned versions, PostgreSQL metadata (Sec 7)
```

## Write a runbook

For each alert, document what it means, how to check, how to re-run or backfill, and who owns it. A short runbook turns a 3 a.m. page into a routine fix.

**Quiz:** Why is a backfill of any past week safe in this design?

- [ ] Data is never stored
- [ ] Airflow forbids backfills
- [x] Tasks are idempotent and use the run's date
- [ ] Because there are no retries

*Answer:* Tasks are idempotent and use the run's date. Deterministic, date-partitioned writes mean re-running a date yields the same result.