Kafka # Queue Configuration

 Quick Answer: In Kafka, queues are implemented using topics and partitions, and their configuration involves setting parameters like replication factor, partitions, retention policies, and consumer group behavior. Unlike traditional message queues, Kafka provides a distributed, fault-tolerant log system where consumers form groups to achieve queue-like semantics.

 

🔑 Kafka Queue Configuration Essentials

1. Topics & Partitions

• Topic → Logical channel where messages are published.
• Partitions → Each topic is split into partitions for scalability and parallelism.
• Configuration:

·       bin/kafka-topics.sh --create \

·         --topic orders \

·         --partitions 3 \

·         --replication-factor 2 \

·         --bootstrap-server localhost:9092

• More partitions = higher throughput, but also more complexity in consumer balancing.

 

2. Replication Factor

• Ensures fault tolerance by replicating partitions across brokers.
• Recommended: replication factor ≥ 3 in production.
• Controlled at topic creation time.

 

3. Retention Policies

• Kafka stores messages for a configurable time, unlike queues that delete after consumption.
• Key configs:
• retention.ms → how long messages are retained.
• retention.bytes → max size before old messages are deleted.
• cleanup.policy → delete (default) or compact (log compaction keeps latest key-value).

 

4. Consumer Groups

• Kafka achieves queue semantics via consumer groups:
• Each message is delivered to one consumer in the group.
• Multiple consumers in the same group share partitions.
• Different groups can consume the same topic independently (pub-sub style).

 

5. Producer Acknowledgements

• Configurable durability guarantees:
• acks=0 → fire-and-forget.
• acks=1 → leader acknowledgment only.
• acks=all → waits for all in-sync replicas (strong durability).

 

6. Other Important Broker Configs

• num.partitions → default partitions per topic.
• default.replication.factor → default replication factor.
• log.segment.bytes → segment size before rolling new log file.
• min.insync.replicas → minimum replicas that must acknowledge for a write to succeed.

 

📌 Example: Queue-like Setup

For a resilient order-processing queue:

• Topic: orders
• Partitions: 6 (to allow parallel consumers)
• Replication factor: 3 (fault tolerance)
• Retention: 7 days (retention.ms=604800000)
• Consumer group: order-service (ensures each order is processed once by one consumer instance).

---

🚀 Real-World Note

In microservices with Spring Boot + Kafka, you configure these via application.yml:

spring:

  kafka:

    consumer:

      group-id: order-service

      auto-offset-reset: earliest

    producer:

      acks: all

      retries: 3

This ensures queue-like delivery semantics with durability and retry handling.

 

In summary: Kafka doesn’t have a “queue” object like RabbitMQ; instead, topics + partitions + consumer groups provide queue semantics. Configuration revolves around replication, retention, and consumer group management.