Lesson 13 / 25

Trigger Rules

Control when a task runs depending on the states of its upstream tasks.

Not only "all succeeded"

By default a task runs when all upstream tasks succeeded (all_success). Other rules include all_done (run once everything finished, whatever the outcome: good for cleanup and notifications), one_failed (run an alert when something failed), none_failed and none_failed_min_one_success (continue after skips, fail if any upstream failed). Choosing the right rule is how you build cleanup steps that always run and alerts that only run on failure.

Evaluating rules, run

I ran a plain-Python model of four rules for upstream states success and failed. Only all_done and one_failed would let the task run; all_success and none_failed would not.

def ok(rule, states):
    s = list(states)
    return {"all_success": all(x == "success" for x in s),
            "all_done": all(x in ("success", "failed", "skipped") for x in s),
            "one_failed": any(x == "failed" for x in s),
            "none_failed": all(x != "failed" for x in s)}[rule]

ups = ["success", "failed"]
print({r: ok(r, ups) for r in ("all_success", "all_done", "one_failed", "none_failed")})

Output:

{'all_success': False, 'all_done': True, 'one_failed': True, 'none_failed': False}

Use all_done for cleanup

Delete temporary tables and files in a task with trigger_rule="all_done" so cleanup runs even after failures.

Quick check: Which rule runs a failure alert only when an upstream task failed?

  • all_success
  • one_failed
  • always_skip
  • none_failed
Answer

one_failed — one_failed triggers as soon as at least one upstream task has failed.