Apache Kafka - Message Storage Architecture

 In Apache Kafka, messages are stored in topics, which are further divided into partitions. Here's how it works:

---

🗃️ Kafka Message Storage Architecture

1. Topics

   • A topic is a logical stream of data.
   • Producers send messages to a topic.
   • Consumers read messages from a topic.

2. Partitions

   • Each topic is split into one or more partitions.
   • Messages within a partition are stored in append-only logs.
   • Each message gets a unique offset — its position in the log.

3. Log Files on Disk

   • Kafka stores messages as log files on disk.
   • These files reside in the Kafka broker’s configured data directory (e.g., /tmp/kafka-logs by default).
   • Each partition has its own set of log files.

4. Retention Policy

   • Messages are retained based on time (e.g., 7 days) or size (e.g., 1 GB).
   • Even after consumption, messages remain until the retention period expires.

5. Replication

   • Kafka replicates partitions across brokers for fault tolerance.
   • One broker acts as the leader, others as followers.

---

📦 Summary

• Messages are stored on disk in partition logs within topics.
• Kafka ensures durability, ordering within partitions, and fault tolerance via replication.

Would you like to see how to configure retention or explore how consumers track offsets?