# Retention and Log Compaction — Apache Kafka: Event Streaming from Basics to Production

Source: https://www.geekswithgeeks.com/en/kafka/td-retention-compaction

> Control how long data is kept and use compaction for "latest value per key" topics.

## Delete by age or keep the latest

With the default **`cleanup.policy=delete`**, old log segments are removed after `retention.ms` (default 7 days) or when a partition exceeds `retention.bytes`. With **`cleanup.policy=compact`**, Kafka keeps at least the **latest record for each key** and eventually removes older ones; a record with a key and a **null value** (a "tombstone") deletes that key. Compaction suits state-like topics (user profiles, configuration, a changelog of a table). Retention is per topic and can be changed at any time with `kafka-configs.sh`. Remember storage = data per day × retention days × replication factor.

## Changing retention and creating a compacted topic, run

I ran these on the real broker. The first sets 24 hours of retention on `orders`; the second creates `user-profile` with `cleanup.policy=compact` and the describe output confirms it.

```bash
kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics \
  --entity-name orders --alter --add-config retention.ms=86400000
kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name orders --describe

kafka-topics.sh --bootstrap-server localhost:9092 --create --topic user-profile \
  --partitions 1 --replication-factor 1 --config cleanup.policy=compact
```

Output:

```
Completed updating config for topic orders.
Dynamic configs for topic orders are:
  retention.ms=86400000 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:retention.ms=86400000}
Created topic user-profile.
```

## What compaction keeps, and storage sizing, run

Compaction keeps the latest value per key and drops keys whose latest value is a tombstone (`None`): here `u2` is deleted and `u1`, `u3` keep their last values. The last line: 40 GB per day, kept 7 days, with 3 replicas needs 840 GB.

```python
from collections import OrderedDict
log = [("u1", "a"), ("u2", "b"), ("u1", "c"), ("u3", "d"), ("u2", None), ("u1", "e")]
latest = OrderedDict()
for k, v in log:
    latest.pop(k, None)
    latest[k] = v
print({k: v for k, v in latest.items() if v is not None})

daily_gb, days, rf = 40, 7, 3
print(daily_gb * days * rf, "GB")
```

Output:

```
{'u3': 'd', 'u1': 'e'}
840 GB
```

**Quiz:** What does a record with a key and a null value do on a compacted topic?

- [ ] Does nothing at all
- [ ] Duplicates the key
- [ ] Deletes the topic
- [x] Marks that key for deletion (a tombstone)

*Answer:* Marks that key for deletion (a tombstone). A tombstone tells compaction to eventually remove that key entirely.