Assignment and Rebalancing

Partitions move when membership changes

When a consumer joins, leaves, crashes (misses heartbeats within session.timeout.ms) or takes too long between polls (max.poll.interval.ms), the group rebalances: partitions are reassigned. The assignor decides how: range and roundrobin are classic; cooperative-sticky keeps most assignments in place and moves only what is needed, avoiding a full stop-the-world pause. Frequent rebalances hurt throughput, so keep processing per poll short, set timeouts sensibly and prefer cooperative assignment. With a 6-partition topic, 2 consumers get 3 partitions each, 6 consumers get one each, and a 7th consumer gets none.

Range assignment, run

I ran a model of the range strategy. Two consumers get 3 partitions each; seven consumers leave the seventh idle; one consumer takes all six.

def range_assign(parts, consumers):
    out = {c: [] for c in consumers}
    per, extra = divmod(len(parts), len(consumers)); i = 0
    for idx, c in enumerate(consumers):
        take = per + (1 if idx < extra else 0)
        out[c] = parts[i:i + take]; i += take
    return out

print(range_assign(list(range(6)), ["c1", "c2"]))
print(range_assign(list(range(6)), ["c1", "c2", "c3", "c4", "c5", "c6", "c7"]))
print(range_assign(list(range(6)), ["c1"]))

Output:

{'c1': [0, 1, 2], 'c2': [3, 4, 5]}
{'c1': [0], 'c2': [1], 'c3': [2], 'c4': [3], 'c5': [4], 'c6': [5], 'c7': []}
{'c1': [0, 1, 2, 3, 4, 5]}

Consumer settings that matter (illustrative)

Typical properties for a stable group.

group.id=billing
auto.offset.reset=earliest
enable.auto.commit=false
max.poll.records=200
max.poll.interval.ms=300000
session.timeout.ms=30000
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

Quick check: A topic has 6 partitions and a group has 8 consumers. What happens?

• 6 consumers get one partition each and 2 sit idle
• All 8 share partitions equally
• The group fails to start
• Kafka creates 2 new partitions automatically