Design a message queue

Clarifying questions

• Is ordering required globally, per key, or not at all?
• Are messages small payloads or references to object storage?
• How long should unconsumed messages be retained?

Functional requirements

• Publish messages to named queues.
• Let consumers lease, acknowledge, and retry messages.
• Expose dead-letter queues and replay controls.

Nonfunctional requirements

• Provide at-least-once delivery with bounded duplicate risk.
• Keep publish latency predictable during consumer outages.
• Prevent one queue from starving other queues.

Scale assumptions

• One million messages per minute at peak.
• Most messages are under 16 KB.
• Some queues are idle for days and then spike.

API sketch

• POST /v1/queues/{name}/messages { bodyRef, orderingKey?, dedupeKey? }
• POST /v1/queues/{name}/lease { maxMessages, leaseMs } -> messages[]

Data model

• messages(queue, partition, offset, body_ref, status, available_at, lease_until).
• consumer_offsets(queue, consumer_group, partition, committed_offset).

Architecture components

• Producers write to partitioned append logs.
• Consumers lease available messages and acknowledge completion.
• A dead-letter policy moves repeatedly failing messages aside.

Bottlenecks

• Hot ordering keys limit partition parallelism.
• Slow consumers cause retention growth and replay lag.

Failure modes

• Consumer crash: lease expires and message becomes visible again.
• Producer retry: dedupeKey prevents duplicate logical messages for a short window.
• Poison message: dead-letter after max attempts with failure reason.

Observability

• Queue depth, oldest visible age, consumer lag, retry count, dead-letter rate.
• Publish and lease latency by queue tier.

Security / privacy

• Authorize producers and consumers per queue.
• Avoid raw PII in message bodies; prefer encrypted payload references.

Cost considerations

• Retention cost grows with consumer lag and body size.
• High fanout may need topic semantics instead of many duplicate queue writes.

Tradeoffs

• Strict ordering simplifies consumers but limits throughput.
• Push delivery reduces polling waste but complicates backpressure.

Rubric