Lec 3: Additional Topics

How Kafka tracks of Consumer offsets?

Kafka keeps track of Consumer Offsets by storing them in a special topic named __consumer_offsets.

Here's a concise example:

When a consumer reads a message from a topic's partition and wants to save its progress, it commits the offset of that message to the __consumer_offsets topic.

ASCII Diagram:

+-------------------------+
|    Topic: user_logs     |
+-------------------------+
| Offset 3: Log Entry C   |
|-------------------------|      +-------------------------------+
| Offset 4: Log Entry D   |----->| __consumer_offsets Topic      |
|-------------------------|      | Consumer: A, Topic: user_logs |
| Offset 5: Log Entry E   |      | Partition: 0, Offset: 4       |
+-------------------------+      +-------------------------------+

In the diagram, the consumer has last read "Log Entry D" from the user_logs topic's partition 0 at offset 4. This progress is saved in the __consumer_offsets topic, which means if the consumer restarts, it knows to resume from offset 5 of user_logs. Some key points:

• __consumer_offsets is a special Kafka topic
• Consumers write their latest offset to this topic
• This allows Kafka to track consumer position
• If a consumer dies, it can resume from the committed offset