Kafka Consumer Groups Explained: Partitions, Rebalancing and Offsets

A Kafka topic is just an append-only log split into partitions. The interesting part is how many applications can read that log at the same time without stepping on each other. That is the job of the consumer group.

What a consumer group is

A consumer group is a set of consumers that share one group.id. Kafka guarantees that, within a group, each partition is read by exactly one consumer at a time.

Two consequences follow from that single rule:

• Consumers in the same group split the work. This is how you scale processing.
• Consumers in different groups each get every message. This is how you fan out to independent applications, for example billing and analytics.

An event on the orders topic might look like this:

{
  "orderId": "7f9c2b1e-4c1a-4a53-9a55-0d7f3a2e9b10",
  "customerId": "c-1042",
  "status": "PLACED",
  "totalCents": 459900,
  "placedAt": "2026-09-24T08:15:30Z"
}

How partitions are assigned

When consumers join a group, one broker acts as the group coordinator and the partitions of the subscribed topics are divided between the members.

The number of partitions sets the ceiling for parallelism. With six partitions:

Consumers in the group	Partitions per consumer	Idle consumers
1	6	0
3	2	0
6	1	0
8	1	2

Adding a seventh consumer to a six-partition topic does not make anything faster. It just sits idle as a standby.

Rebalancing

A rebalance is the process of redistributing partitions across the members of a group.

What triggers a rebalance

• A consumer joins the group, for example during a deployment or a scale-up.
• A consumer leaves cleanly by calling close().
• A consumer is considered dead because it stopped sending heartbeats within session.timeout.ms.
• A consumer takes longer than max.poll.interval.ms between two calls to poll().
• The number of partitions on a subscribed topic changes.

The fourth one causes most production incidents. If processing one batch takes longer than max.poll.interval.ms (five minutes by default), the consumer is removed from the group, its partitions move elsewhere, and the same slow batch is processed again by another consumer.

Cooperative rebalancing

With the older eager protocol every consumer gives up all its partitions at the start of a rebalance, so the whole group stops. With the cooperative protocol only the partitions that actually need to move are revoked, and everyone else keeps working.

For the classic consumer you opt in with the CooperativeStickyAssignor, as shown in the Java example below.

Offsets and delivery guarantees

Each group stores, per partition, the offset of the next message it should read. This lives in the internal __consumer_offsets topic. When you commit decides what happens after a crash:

1. Commit after processing gives you at-least-once delivery. A crash between processing and committing means the batch is processed again, so your handler must be idempotent.
2. Commit before processing gives you at-most-once delivery. A crash after the commit means those messages are lost.

Auto-commit (enable.auto.commit=true) commits on a timer during poll(), which makes the guarantee depend on timing. For anything that matters, turn it off and commit yourself.

A minimal Java consumer

import java.time.Duration;
import java.util.List;
import java.util.Properties;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.CooperativeStickyAssignor;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;

public class OrderConsumer {

    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "order-processor");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
        props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
                CooperativeStickyAssignor.class.getName());

        try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
            consumer.subscribe(List.of("orders"));

            while (true) {
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
                for (ConsumerRecord<String, String> record : records) {
                    process(record);
                }
                // At-least-once: commit only after the whole batch was processed.
                consumer.commitSync();
            }
        }
    }

    private static void process(ConsumerRecord<String, String> record) {
        System.out.printf("partition=%d offset=%d key=%s%n",
                record.partition(), record.offset(), record.key());
    }
}

Start a second copy of this program and you will see the partitions split between the two processes within a few seconds.

Configuration that matters

The same settings in a Spring Boot application.yml:

spring:
  kafka:
    bootstrap-servers: localhost:9092
    consumer:
      group-id: order-processor
      auto-offset-reset: earliest
      enable-auto-commit: false
      max-poll-records: 200
      properties:
        partition.assignment.strategy: org.apache.kafka.clients.consumer.CooperativeStickyAssignor
        max.poll.interval.ms: 300000
        session.timeout.ms: 45000
    listener:
      ack-mode: batch

If your handler is slow, lower max-poll-records before you raise max.poll.interval.ms. Smaller batches keep the consumer inside the poll interval without hiding a stuck consumer for longer.

Checking lag from the command line

Lag is the distance between the newest offset in a partition and the offset your group has committed. It is the single most useful number for a consumer.

bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --describe \
  --group order-processor

The output lists CURRENT-OFFSET, LOG-END-OFFSET and LAG for every partition, along with the consumer that currently owns it. Lag that keeps growing means you need faster processing, more partitions and consumers, or both.

Summary

• One partition is read by one consumer per group, so partitions cap your parallelism.
• Rebalances happen when members join, leave or stop polling in time.
• Use cooperative rebalancing so a deployment does not pause the whole group.
• Commit after processing and make handlers idempotent.
• Watch consumer lag.