Priya Nair·
Kafka cluster config plus consumer code that commits after the side effect, dedupe and DLQ included
Compares and designs message queue infrastructure with topic partitioning, consumer group strategies, and exactly-once processing guarantees.
Message Queue Architecture & Selection
You are a distributed messaging architect who designed queue systems processing trillions of messages. Design a message queue architecture.
**Message Volume**: {{message_volume}} (messages per second, peak burst capacity needed)
**Message Types**: {{message_types}} (event notifications, job queues, command messages, audit logs)
**Delivery Semantics**: {{delivery_semantics}} (at-most-once, at-least-once, exactly-once)
**Ordering Requirements**: {{ordering_requirements}} (global order, per-entity order, no ordering needed)
Provide:
1. **Queue Technology Comparison** - Apache Kafka, RabbitMQ, NATS Streaming/JetStream, AWS SQS/SNS, Apache Pulsar, Redis Streams scored across 10 dimensions
2. **Cluster Architecture** - Broker count, replication factor (RF=3), min.insync.replicas, rack-awareness, controller election
3. **Topic Design** - Topic naming convention, partition count formula (max(throughput/consumer_rate, consumer_count)), partition key selection
4. **Producer Configuration** - Batching (linger.ms, batch.size), compression (lz4/snappy/zstd), acks=all, idempotent producer, transactional producer
5. **Consumer Architecture** - Consumer group design, partition assignment strategy (range/round-robin/sticky/cooperative), offset management
6. **Exactly-Once Processing** - Idempotent producer + transactional consumer configuration with transaction boundaries
7. **Schema Management** - Schema Registry (Avro/JSON Schema/Protobuf), compatibility modes (backward/forward/full), evolution rules
8. **Dead Letter Queches** - DLQ per topic, retry topics with backoff (exponential delay), max retry configuration, alerting
9. **Monitoring Setup** - Consumer lag alerts, broker disk usage, under-replicated partitions, offline partitions, request latency
10. **Security** - SASL/SSL configuration, ACLs per topic, encryption at rest and in transit, credential rotation
11. **Disaster Recovery** - MirrorMaker 2.0 for cross-cluster replication, backup strategy, failover procedures
12. **Performance Tuning** - OS-level tuning (ulimit, vm.swappiness, disk scheduler), JVM tuning for Kafka, network tuning
13. **Migration Plan** - Strategy for migrating from existing queue to new without message loss or downtime
Include actual Kafka configuration (server.properties) and producer/consumer code examples.
Ergebnisse
Message-queue design for **order + audit events** at ~50k msg/s peak, per-entity ordering, at-least-once with idempotent consumers. **Kafka wins** here — partitioned ordering plus replay is exactly the requirement.
**Why not the others (short version):** RabbitMQ tops out earlier and lacks cheap replay; SQS gives no per-key ordering without FIFO's throughput cap; Kafka's log + consumer groups fit cleanly.
**Cluster:** 3 brokers, `replication.factor=3`, `min.insync.replicas=2`, rack-aware. **Topic:** `orders.events`, 24 partitions keyed on `orderId` so all events for one order stay ordered.
```properties
# producer
acks=all
enable.idempotence=true
compression.type=lz4
linger.ms=10
batch.size=65536
```
```typescript
// consumer — commit AFTER the side effect, dedupe on offset key
await processor.run(async (msg) => {
if (await seen(msg.key)) return; // idempotency guard
await applyEffect(JSON.parse(msg.value));
await markSeen(msg.key, "24h");
await consumer.commitOffsets([{ topic, partition: msg.partition,
offset: (Number(msg.offset) + 1).toString() }]);
});
```
**DLQ:** failures route to `orders.events.dlq` after retry topics with escalating delay; alert on DLQ depth > 0. **Monitoring:** consumer lag, under-replicated partitions, and `offline_partitions_count` are the three alerts that catch real incidents. Cross-cluster DR via MirrorMaker 2.0.
Modell: Claude Opus 4
25 Likes8 SavesScore: 21
2 Kommentare
Jonas Weber·
The Big-O note at the end sold me.
Luca Brunner·
Did not expect a system design prompt to be this thorough.