Idempotent Producers and Transactions

Exactly-once inside Kafka

Kafka's transactions let a producer write to several partitions and commit consumer offsets atomically: either all of it becomes visible or none of it. A consume-process-produce application can therefore read from topic A and write to topic B with exactly-once semantics within Kafka: set a transactional.id, wrap the work in begin/commit, and have downstream consumers use isolation.level=read_committed so they never see aborted data. The guarantee covers Kafka reads and writes; if you also update an external database, you still need an idempotent write or a transactional outbox, because Kafka cannot roll back systems it does not control. Frameworks such as Kafka Streams and Flink can provide end-to-end exactly-once for supported sinks.

A transactional producer (illustrative)

Reads from orders, writes to orders-enriched and commits offsets in one transaction. enrich stands for your own function.

p = Producer({"bootstrap.servers": "localhost:9092", "transactional.id": "enricher-1"})
c = Consumer({"bootstrap.servers": "localhost:9092", "group.id": "enricher",
              "enable.auto.commit": False, "isolation.level": "read_committed"})
p.init_transactions()
c.subscribe(["orders"])
while True:
    msg = c.poll(1.0)
    if msg is None or msg.error(): continue
    p.begin_transaction()
    p.produce("orders-enriched", key=msg.key(), value=enrich(msg.value()))
    p.send_offsets_to_transaction(c.position(c.assignment()), c.consumer_group_metadata())
    p.commit_transaction()

Quick check: What does `isolation.level=read_committed` do for a consumer?

• Hides records from aborted or open transactions
• Encrypts the records
• Deletes old records
• Speeds up the broker