Kafka Fundamentals: Partitions, Consumer Groups, and Ordering

The mistake almost everyone makes with Kafka — I made it too — is thinking of it as a message queue like RabbitMQ. It isn't. Kafka is a distributed, append-only commit log, and once that clicks, partitions, consumer groups, and its ordering guarantees stop being trivia to memorize and start being obvious consequences of the design.

A log, not a queue

A traditional queue deletes a message once a consumer acknowledges it. Kafka doesn't. Producers append records to the end of a log; consumers read from wherever they choose and track their own position. The data stays put, based on a retention policy, whether or not anyone read it.

This is why Kafka can do things a queue can't: multiple independent consumers can read the same stream at their own pace, and you can replay history by resetting a consumer's position back to the start.

Topics and partitions: the unit of parallelism

A topic is a named stream. But a topic is split into partitions, and the partition is where all the interesting behavior lives. Each partition is an independent, ordered log.

Topic "orders"
  Partition 0:  [o1] [o4] [o7] ...   ← ordered within the partition
  Partition 1:  [o2] [o5] [o8] ...
  Partition 2:  [o3] [o6] [o9] ...

Partitions are how Kafka scales: they're spread across brokers, and different consumers read different partitions in parallel. More partitions means more possible parallelism. This is the single most important number you pick, and it's hard to reduce later — so lean slightly high.

Ordering: guaranteed, but only within a partition

Here's the rule that trips people up: Kafka guarantees order within a partition, and nowhere else. Records in partition 0 are read in the exact order they were written. But there is no global ordering across partitions.

So if order matters for related events, they must land in the same partition. That's what the message key controls: Kafka hashes the key to pick a partition, so all records with the same key go to the same partition and stay ordered relative to each other.

// Same key → same partition → ordered. Here, all events for one order stay in sequence.
producer.send(new ProducerRecord<>("orders", order.getId(), event));

Consumer groups: how work gets shared

A consumer group is a set of consumers that cooperate to read a topic. Kafka assigns each partition to exactly one consumer within a group. That gives you a clean scaling model:

• More consumers in a group → partitions spread across them → more throughput.
• But you can never have more useful consumers than partitions. With 3 partitions and 4 consumers, one consumer sits idle. Partition count is your parallelism ceiling.

Meanwhile, different groups are independent. The analytics group and the notification group both read every record, each tracking its own position. Same stream, many independent readers — that's the log model paying off.

Topic "orders" (3 partitions)
 
Group "billing"   → 3 consumers, one partition each (max parallelism)
Group "analytics" → 1 consumer reading all 3 (its own independent position)

Offsets and delivery guarantees

Each consumer commits an offset — its position in each partition. Where you commit decides your delivery semantics:

• Commit after processing → at-least-once. Crash mid-process and you'll reprocess on restart. This is the sane default, and it's why consumers must be idempotent.
• Commit before processing → at-most-once. Crash and you lose the record. Rarely what you want.

Exactly-once exists via transactions, but it's narrower and costlier than it sounds. For most systems, the right posture is at-least-once delivery plus idempotent consumers, so a reprocessed record is harmless.

When Kafka is the right call

Kafka shines when you need a durable, replayable stream that many independent services consume — event sourcing, analytics pipelines, decoupling services through events. It's overkill for simple background jobs; for "run this task later," a Redis-backed queue is lighter and simpler. Reach for Kafka when the log itself — its order, its retention, its replayability — is the thing you actually need.