Case Study: Order Events for an E-commerce Platform

The design

Topic sales.order.placed.v1: key = order_id (all events of an order stay ordered), 12 partitions (from the throughput calculation with headroom), replication factor 3, min.insync.replicas=2, retention 7 days. Schema in Avro with a registry and backward compatibility. The order service produces with acks=all, idempotence on, linger.ms=10, lz4. Three consumer groups read independently: billing (manual commit after an idempotent database write, DLT for bad records), fulfilment (cooperative-sticky assignor), analytics (a sink connector to the lake). Monitoring covers lag per group, under-replicated partitions and DLT size. Access is by ACL per service, TLS everywhere, and topic changes go through code review.

A reliable order-events pipeline

Key choice, durable settings, idempotent consumers, monitoring and security combine into a dependable design.

The design on one page

Each line maps to a section of this course.

Topic        key=order_id, 12 partitions, rf=3, retention 7d     (Sec 1, 4)
Durability   min.insync.replicas=2, acks=all, idempotence, no unclean  (Sec 2, 5)
Schema       Avro + registry, backward-compatible                  (Sec 2)
Consumers    billing (manual commit + DLT), fulfilment, analytics   (Sec 3, 6)
Ops          lag/URP/DLT alerts, TLS + SASL + ACLs, rolling upgrades (Sec 7)

Load-test before launch

Replay a realistic volume into a staging cluster, kill a broker and a consumer during the test, and confirm lag recovers and no records are lost or double-applied.

Quick check: Why is `order_id` a good key in the case study?

• It disables compression
• It makes every record unique across topics
• All events of one order go to one partition and stay ordered
• Kafka requires it