Your First Topic: Create, Produce, Consume

Try it locally

The quickest way to experiment is the official Docker image, which starts a single-node broker in KRaft mode. The tools kafka-topics.sh, kafka-console-producer.sh and kafka-console-consumer.sh live in the image under /opt/kafka/bin. You create a topic with a partition count, produce lines (optionally with keys), and consume them from the beginning. This is for learning only; production uses several brokers.

Start a broker and create a topic, run

I ran these against Kafka 3.9.0. The container image needs about ten seconds to be ready.

docker run -d --name kafka-demo apache/kafka:3.9.0
docker exec kafka-demo /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic orders --partitions 6 --replication-factor 1
docker exec kafka-demo /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

Output:

Created topic orders.
orders

Produce keyed records and read them back, run

Each line is key:value. The consumer prints the partition and offset; note that user-1 and user-2 landed in partition 2 and the two user-1 records are in order (offsets 0 and 2). The list is sorted for readability.

printf 'user-1:placed\nuser-2:placed\nuser-3:placed\nuser-4:placed\nuser-5:placed\nuser-6:placed\nuser-1:paid\n' | \
  docker exec -i kafka-demo /opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server localhost:9092 --topic orders \
  --property parse.key=true --property key.separator=:

docker exec kafka-demo /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 --topic orders --from-beginning --max-messages 7 \
  --property print.key=true --property print.partition=true --property print.offset=true

Output:

Partition:1	Offset:0	user-4	placed
Partition:2	Offset:0	user-1	placed
Partition:2	Offset:1	user-2	placed
Partition:2	Offset:2	user-1	paid
Partition:4	Offset:0	user-5	placed
Partition:5	Offset:0	user-3	placed
Partition:5	Offset:1	user-6	placed

Quick check: In the output, both `user-1` records are in the same partition. Why?

• Kafka picks partitions randomly each time
• The same key is hashed to the same partition
• Partitions are chosen alphabetically by value
• It was a coincidence of the broker