Message Queues & Asynchronous Processing

Distributed Systems & Scalability

Chapter 6 · Message Queues & Asynchronous Processing

Software Architecture Fundamentals Chapter 6 built an EventBus that dispatched to subscribers the instant publish() was called — decoupling who knows about whom, but not when the work happens. A message queue decouples both. This chapter verifies what that second kind of decoupling actually buys, and what it costs.

Real Buffering: Decoupling Timing, Not Just Knowledge

class MessageQueue:
    def __init__(self): self.messages = []
    def publish(self, message): self.messages.append(message)
    def consume_all(self, handler):
        while self.messages:
            message = self.messages.pop(0)
            handler(message)
    

Verified directly — publishing works with no consumer running at all

Publishing 5 messages with no call to consume_all() anywhere yet leaves all 5 correctly sitting in queue.messages, genuinely unprocessed. Starting the consumer afterward correctly processes all 5, in the exact order they were published, and drains the queue to empty. Compare this to Software Architecture Fundamentals' own EventBus.publish(), which had no way to "publish" without a subscriber already attached to react immediately — this queue lets production and consumption happen on completely independent schedules.

At-Least-Once Delivery: Why Idempotency Isn't Optional

A message that's processed but never acknowledged — because the consumer crashed in between — gets redelivered. What happens depends entirely on whether the handler can safely run twice.

Verified directly — a non-idempotent handler processes the same message twice, and the bug is real

Publishing one message worth 10 points, then simulating a crash after processing but before acknowledging it: the handler already ran once (points = 10), but the message stays in the queue since it was never acked. Redelivery correctly re-triggers the handler — and points becomes 20, not the correct 10. The same real award was granted twice for one real event.

Verified directly — an idempotent handler processes the identical redelivery safely

The identical crash-then-redeliver scenario, but the handler checks a processed_ids set before doing anything: points correctly stays at 10 after both the crashed first attempt and the successful redelivery. The message was still delivered twice — at-least-once delivery didn't change — but the handler's own idempotency absorbed the duplicate safely.

"At-least-once" is a promise about delivery, not about correctness

A message queue guaranteeing at-least-once delivery is explicitly promising a message might arrive more than once — it is not promising your handler will behave correctly when that happens. The 20-instead-of-10 bug just verified is not a queue malfunction; the queue did exactly what "at-least-once" means. The correctness burden sits entirely with the consumer.

Dead-Letter Queues: What Happens When a Message Can't Be Processed

Verified directly — an always-failing message gets retried, then moved aside, not retried forever

A message that raises an exception every time it's processed, against a queue configured with max_retries=3: the first two attempts return 'requeued', the third returns 'dead-lettered' — the message moves into dead_letter_queue, carrying the actual error message ('malformed payload: missing required field'), and stops consuming further processing attempts entirely.

Verified directly — a transient failure gets a real chance to recover, and never gets dead-lettered

A handler that fails on its first two attempts but succeeds on the third: the outcomes were ['requeued', 'requeued', 'success'] — dead_letter_queue stayed empty. The retry mechanism gave the same message multiple genuine chances, and a transient problem (the kind Chapter 9's own resilience-pattern chapter covers in depth) resolved itself without ever needing manual intervention.

Where This Connects

This chapter's finding	What it connects to
Genuine timing decoupling, verified against Software Architecture Fundamentals' synchronous `EventBus`	Software Architecture Fundamentals Chapter 6's own eventual-consistency window — a message queue is the deliberate, controlled version of that same gap
A real, verified double-processing bug from non-idempotent handling	Chapter 5's own AP reconciliation finding — both are "the same real event counted twice," at different layers
Dead-lettering only after genuine retries, verified not triggering on a recoverable transient failure	Chapter 9's own Fault Tolerance & Resilience Patterns — retry-with-backoff and dead-letter queues are close cousins, covered together there

Hands-On Exercises

Exercise 1

Using this chapter's own MessageQueue, publish 3 messages, consume just 1 of them (call consume_all won't work here — write a small loop that pops and handles only 1), then publish 2 more messages before finally consuming the rest. Verify the final processing order is correct.

📄 View solution

Exercise 2

Using this chapter's own AtLeastOnceQueue and idempotent handler pattern, simulate the message crashing twice before finally being acknowledged successfully (three total delivery attempts). Verify points still correctly ends at 10, not 30.

📄 View solution

Exercise 3

Using this chapter's own two verified findings (the double-processing bug and the dead-letter queue), explain why a message ending up in the dead-letter queue is not a case at-least-once delivery's own idempotency requirement can help with — what's the actual difference in what's failing?

📄 View solution

Chapter 6 Quick Reference

Message queues: buffer messages between producer and consumer — verified: 5 messages queued and stayed unprocessed with zero consumers running, then processed correctly, in order, once one started
At-least-once, verified: a non-idempotent handler double-counted a redelivered message (10 → 20); an idempotent one, tracking processed IDs, correctly stayed at 10
Dead-letter queues, verified: an always-failing message was retried twice then moved aside on the third attempt; a transient failure recovered on its own within the same retry budget and was never dead-lettered
Next chapter: Rate Limiting & Throttling — protecting a system's own downstream services from being overwhelmed in the first place