Failures, Retries and Dead-Letter Topics

One bad record must not stop the line

A partition is processed in order, so a record that always fails (a poison message: bad data, unexpected schema) can block everything behind it if you retry forever. Separate transient failures (a database timeout, which deserve a few retries with backoff) from permanent ones (malformed data, which retries cannot fix). Send permanent failures to a dead-letter topic (DLT) together with the error reason and original coordinates (topic, partition, offset), commit the offset, and continue. Monitor the DLT size and have a process to inspect and replay its records after fixing the cause. Where ordering matters, retry in place rather than moving the record elsewhere.

Routing a failure to a dead-letter topic (illustrative)

TransientError is retried by your code; anything else is considered permanent and parked with context in orders.dlt.

try:
    handle(msg)
except TransientError:
    raise                                  # let the retry policy handle it
except Exception as exc:                   # permanent: park it, keep going
    producer.produce("orders.dlt", key=msg.key(), value=msg.value(),
                     headers=[("error", str(exc).encode()),
                              ("src", f"{msg.topic()}[{msg.partition()}]@{msg.offset()}".encode())])
commit(msg)

Quick check: Why use a dead-letter topic?

• To speed up the broker
• To park permanently failing records so the partition keeps moving
• To delete all errors silently
• To increase partitions