Apache Kafka — event streaming for microservices

RabbitMQ serves us well for task queues. But for event-driven architecture — where you want every interested party to receive an event and be able to replay it — you need a different model. Apache Kafka is a distributed commit log, not a message queue.

Kafka vs. RabbitMQ¶

RabbitMQ: a message is delivered to one consumer and deleted. Kafka: a message is written to the log and stored (days, weeks). Each consumer reads from the log independently. Replay is possible.

Concepts¶

Topics: Logical channels (order-events, user-events). Partitions: Horizontal scaling — a topic is split into partitions. Consumer groups: Consumers in a group share partitions. Offsets: Each consumer remembers where it left off in the log.

// Producer
Properties props = new Properties();
props.put("bootstrap.servers", "kafka:9092");
props.put("key.serializer", StringSerializer.class);
props.put("value.serializer", StringSerializer.class);

Producer<String, String> producer = new KafkaProducer<>(props);
producer.send(new ProducerRecord<>("order-events", orderId, orderJson));

Use cases¶

• Event sourcing — storing events instead of state
• Data pipeline — streaming data between systems
• Log aggregation — centralized log collection
• Stream processing — real-time data transformation

Operational experience¶

Kafka requires ZooKeeper for coordination. A cluster with 3 brokers and replication factor 3 is the minimum for production. More operationally demanding than RabbitMQ, but throughput is orders of magnitude higher.

Kafka for events, RabbitMQ for tasks¶

Kafka is not a replacement for RabbitMQ — it is a different tool for a different problem. Event-driven architecture with Kafka, task queues with RabbitMQ. Both in our stack.