Retention and Log Compaction

Delete by age or keep the latest

With the default cleanup.policy=delete, old log segments are removed after retention.ms (default 7 days) or when a partition exceeds retention.bytes. With cleanup.policy=compact, Kafka keeps at least the latest record for each key and eventually removes older ones; a record with a key and a null value (a "tombstone") deletes that key. Compaction suits state-like topics (user profiles, configuration, a changelog of a table). Retention is per topic and can be changed at any time with kafka-configs.sh. Remember storage = data per day × retention days × replication factor.

Changing retention and creating a compacted topic, run

I ran these on the real broker. The first sets 24 hours of retention on orders; the second creates user-profile with cleanup.policy=compact and the describe output confirms it.

kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics \
  --entity-name orders --alter --add-config retention.ms=86400000
kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name orders --describe

kafka-topics.sh --bootstrap-server localhost:9092 --create --topic user-profile \
  --partitions 1 --replication-factor 1 --config cleanup.policy=compact

Output:

Completed updating config for topic orders.
Dynamic configs for topic orders are:
  retention.ms=86400000 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:retention.ms=86400000}
Created topic user-profile.

What compaction keeps, and storage sizing, run

Compaction keeps the latest value per key and drops keys whose latest value is a tombstone (None): here u2 is deleted and u1, u3 keep their last values. The last line: 40 GB per day, kept 7 days, with 3 replicas needs 840 GB.

from collections import OrderedDict
log = [("u1", "a"), ("u2", "b"), ("u1", "c"), ("u3", "d"), ("u2", None), ("u1", "e")]
latest = OrderedDict()
for k, v in log:
    latest.pop(k, None)
    latest[k] = v
print({k: v for k, v in latest.items() if v is not None})

daily_gb, days, rf = 40, 7, 3
print(daily_gb * days * rf, "GB")

Output:

{'u3': 'd', 'u1': 'e'}
840 GB

Quick check: What does a record with a key and a null value do on a compacted topic?

• Does nothing at all
• Duplicates the key
• Deletes the topic
• Marks that key for deletion (a tombstone)