Messaging Systems

Every large system eventually learns the same lesson: not all work deserves to make the user wait.

At the end of Part 6, we left a question hanging: when data changes, how do invalidation events actually travel through a large system? This article answers that question — and a much bigger one hiding behind it.

Let's start with a scene. A user taps "Place Order" on a food delivery app. Behind that single tap, a small storm begins: the restaurant must be notified, a rider search must start, an invoice must be generated, an email and an SMS must go out, loyalty points must be added, analytics must record the order, the search index must update.

Here's the question that changes how you design systems: should the user wait for all of that before seeing "Order Confirmed"? Of course not. The confirmation must be instant. Everything else can happen soon.

And the moment you say the word soon, you have left the world of synchronous APIs. Welcome to messaging.

The Wall: Where Synchronous Thinking Breaks

So far in this series — Parts 4, 5, and 6 — we mostly followed the read path: a request goes in, a response comes out, caches make it fast. Request-response is a beautiful model. You ask. You wait. You get an answer.

Now watch what happens when one action triggers many operations, and we wire it all synchronously:

User taps "Place Order"
   ↓
Order API
   ↓ call Notification Service   (wait…)
   ↓ call Invoice Service        (wait…)
   ↓ call Loyalty Service        (wait…)
   ↓ call Analytics Service      (wait…)
   ↓ call Search Indexer         (wait…)
   ↓
Finally respond to user

Two problems appear immediately.

Problem 1: Latency stacks up. Five downstream calls at 200 ms each — the user now waits over a second for work they never needed to wait for.

Problem 2: Failures multiply. In a synchronous chain, everyone must be alive at the same moment. If each downstream service is 99.9% available, a chain of five gives the user roughly:

0.999 × 0.999 × 0.999 × 0.999 × 0.999 ≈ 99.5%

Your order flow is now less reliable than its weakest dependency, because it depends on all of them, simultaneously. And notice the worst part: should an order fail because the loyalty-points service is down? Loyalty points have nothing to do with whether the order is valid. Yet in a synchronous chain, they can take the whole flow down with them.

This is the real disease, and it has a name: coupling. The latency is annoying. The coupling is dangerous. When every downstream operation must finish before the user gets a response, the system becomes slow, fragile, and — worst of all — difficult to evolve. Adding one more downstream step means touching, testing, and risking the main flow, every time.

Part 2 taught us that failure is normal in distributed systems. Synchronous chains ignore that lesson. Messaging systems are built around it.

The Fix: Put a Middleman in the Middle

The core idea of messaging is almost embarrassingly simple: instead of calling someone directly, leave a message. Let them handle it when they can.

Producer  →  Broker  →  Consumer
(creates      (stores &     (picks up
 the work)     delivers)     the work)

The producer creates a message: "Order #123 was placed." The broker — the messaging system — stores it safely. The consumer picks it up and does the work.

The Kitchen Rail

Think of a busy restaurant kitchen. The waiter doesn't grab a cook and stand there while the biryani is made. The waiter clips an order ticket onto the rail and walks away. Cooks pick tickets off the rail when their hands are free. If one cook steps away, another picks up the next ticket. If ten orders arrive in one minute, the rail holds them — the kitchen doesn't explode.

That rail is a message broker.

What the Middleman Buys You

Temporal decoupling. The producer and consumer no longer need to be alive at the same moment. The notification service can be down for two minutes; the messages wait patiently.

Load leveling. Part 6 showed how caches absorb read spikes. Brokers absorb work spikes. A flash-sale burst of 50,000 orders becomes a backlog that consumers drain at their own sustainable pace.

Independent scaling. Invoice generation is slow? Add more invoice consumers. Nobody else changes.

Decoupled evolution. This is the quiet superpower. The order service says "Order #123 was placed" and doesn't know or care who's listening. Next month, a fraud-check service starts listening too. The order service is not touched.

Remember write-behind caching from Part 6 — "write to cache now, flush to the database later"? This is the machinery it depends on. That pending write is a message. Which is why, later in this article, delivery guarantees will matter so much.

Events vs Commands: Two Kinds of Messages

Before going deeper, one distinction that saves years of confusion. Every message is one of two things.

A command: "Generate the invoice for order #123." Imperative — do this. Directed at one specific owner. The sender expects the action to happen.

An event: "Order #123 was placed." Past tense — this happened. A statement of fact. Zero, one, or twenty services may react — the publisher doesn't know.

A quick heuristic that works surprisingly well: if renaming the message to past tense feels wrong, it's a command.

Commands centralize intent. Events broadcast facts. Mature systems use both — but architects who confuse them end up publishing "events" like "UpdateSearchIndex" that secretly order one service around, recreating coupling with extra steps.

Queue vs Stream: The Difference Most Engineers Miss

Ask ten engineers the difference between a queue and a stream. Eight will say: "Streams are… faster? Bigger? Kafka?" Here's the distinction that actually matters: a queue is about task distribution. A stream is about event history.

The Queue: A To-Do List

A queue holds work to be done.

Producer → [ msg1 | msg2 | msg3 ] → Consumers compete
                                     ↓
                          One consumer takes msg1
                                     ↓
                          msg1 is GONE from the queue

Consumers compete for messages. Each message is processed by one consumer. Once processed, the message is gone. Perfect when work must be done, once, by someone.

Queue-shaped jobs: send an email notification, process an image upload, generate an invoice PDF. Nobody needs the "send email" task to exist after the email is sent.

The Stream: A Ledger

A stream holds things that happened. Think of a shop's daily sales register — every sale is written down, in order, and the entry is never erased. The accountant reads it, the auditor reads it, the owner reads it. Each of them reads the same ledger, independently, at their own pace.

Producer → [ e1 | e2 | e3 | e4 | e5 | ... ]   (append-only log)
                ↑              ↑
        Analytics is       Fraud service is
        at position 2      at position 5

Events are not deleted when read. Each consumer tracks its own position. Events are appended, retained, and replayable. Multiple independent consumers each receive everything. A brand-new consumer can start today and replay last month's events.

Stream-shaped facts: order placed, payment completed, user activity events, audit events.

The superpower: replay. Six months from now, someone builds a new recommendation service. With request-response, it starts from zero knowledge. With a stream, it replays six months of OrderPlaced events and arrives already educated. You cannot replay an API call that happened last Tuesday. You can replay an event.

Side by Side

Neither is "better." They answer different questions: a queue asks "who will do this work?" A stream asks "what happened, and who needs to know?"

Point-to-Point vs Pub/Sub

Related, but a different axis: how many parties hear the message?

Point-to-point: one producer, one logical consumer — the classic queue.

Order Service → [invoice-queue] → Invoice Workers

Publish/Subscribe: publish once, every subscriber gets a copy.

                       ┌→ Notification Service
Order Service → OrderPlaced ─→ Loyalty Service
   (publishes once)    ├→ Analytics
                       └→ Search Indexer

And here is Part 6's cliffhanger, resolved:

Restaurant edits menu
   ↓
MenuUpdated event published
   ├→ Cache Invalidator  → deletes menu:123 from Redis
   ├→ CDN Purger         → purges /restaurant/123
   ├→ Search Indexer     → reindexes the restaurant
   └→ Analytics          → records the change

That is how invalidation events travel through a large system. The menu service doesn't know the cache exists. It just announces what happened.

Consumer Groups: Compete Inside, Broadcast Outside

Streams give you both behaviors at once, through consumer groups. Within a group, consumers split the work — they compete, like a queue. Across groups, every group independently receives everything.

              OrderPlaced stream
              ↓              ↓
   ┌── Group: invoices ──┐   ┌── Group: analytics ──┐
   │  worker1  worker2   │   │      worker1         │
   │  (split the load)   │   │  (gets everything)   │
   └─────────────────────┘   └──────────────────────┘

Three invoice workers share the load. Analytics, in its own group, still sees every single event. One log. Queue-like scaling inside each team. Broadcast across teams.

Message Ordering: Cheaper Than You Think, Costlier Than You Hope

A natural worry: if everything is async, what keeps events in order? First, the honest answer: global ordering across a large system is expensive, and usually unnecessary.

The practical answer is per-key ordering: all events for the same entity stay in order; events for different entities don't need to. OrderPlaced must come before OrderCancelled for order #123. Whether order #123's events interleave with order #456's? Nobody cares. Streams achieve this by routing all events with the same key (order ID) to the same partition, which preserves order within it.

Ask "ordered per what?" — not "ordered?"

When Things Fail: Retries and Dead-Letter Queues

Part 2's rule applies with full force here: failure is normal. A consumer picks up a message, calls the email provider, and crashes. What happens to the message? It gets redelivered — the broker notices the work was never confirmed and hands it to another consumer. This is the first safety net, but it needs company.

Retry strategies: immediate retry (fine for one-off blips), delayed retry (wait a bit, maybe the provider recovers), exponential backoff (wait 1s, then 2s, then 4s — don't hammer a struggling dependency). These patterns, and their sharp edges, get a full treatment in Part 22: Designing for Failure.

The poison message. Now imagine a message that fails every time — a malformed payload, a bug, a permanently invalid state. Retry it forever and it clogs the pipeline while healthy messages pile up behind it. The answer is the dead-letter queue (DLQ): after N failed attempts, the message is moved aside into a separate parking area where humans and alerts can inspect it.

Main queue → fail → retry → fail → retry → fail
                                      ↓
                              Dead-Letter Queue
                              (alert the owners)

One rule, in the spirit of Part 6's ownership rule: a DLQ nobody monitors is just a slower way to lose messages.

Delivery Guarantees Are Business Decisions

Here's where messaging stops being an infrastructure topic and becomes an architecture topic. Every messaging system must answer: if things go wrong, would you rather lose a message or see it twice? You cannot fully have both. Pick your poison, per workflow.

At-most-once. Fire and forget, no retries. May lose messages, never duplicates. Acceptable for a metrics ping, a presence blip, a "user is typing…" signal — losing one is invisible.

At-least-once. Retry until confirmed. Never loses, may duplicate — a consumer that crashed after doing the work but before confirming will cause a redelivery. This is the workhorse guarantee of serious systems, and it has a visible symptom you've experienced: the duplicate email, the double SMS. That's at-least-once delivery meeting non-idempotent code.

What about exactly-once? The most misunderstood phrase in messaging. Within a single platform's boundaries, some systems can get close under specific configurations. But end-to-end exactly-once across arbitrary systems — "send this email exactly once, ever, no matter what crashes" — is not something you should design your business around.

The practical grown-up answer is effectively-once: at-least-once delivery plus idempotent processing. Make the processing safe to repeat:

On message "charge order #123":
   Have I already processed message id M-9981?
      Yes → acknowledge, do nothing
      No  → charge, record M-9981, acknowledge

Same message, delivered twice, charged once.

Two principles to tattoo somewhere visible: duplicate handling belongs in business logic, not only in infrastructure — the broker can promise redelivery, but only your code knows what "already done" means. And the right guarantee is a business decision — losing an analytics event costs a rounding error, losing a payment event costs money and trust, double-sending an OTP annoys, double-charging enrages.

We'll go deep on idempotency — keys, storage, and the patterns that make payments safe — in Part 21: Payments and Transactions. (That's also where multi-step business transactions across services, sagas, get their treatment.)

Backpressure: When Producers Outrun Consumers

A queue absorbs bursts. But what if the burst never ends? Producers writing 10,000 messages/second, consumers processing 4,000/second — the backlog grows by 6,000 every second, like a sink filling faster than the drain empties. Eventually something gives: memory, disk, or your delivery-time promise ("your invoice will arrive within a minute" quietly becomes an hour).

The vital sign to watch is consumer lag — how far behind consumers are. The honest responses, roughly in order: scale consumers (more cooks at the rail), throttle producers (slow the intake, make some callers wait), shed load (drop what's droppable, at-most-once traffic first), and bound the queue (refuse new work loudly instead of failing silently later).

A queue that only ever grows isn't buffering. It's postponing a crash.

Eventual Consistency: The Price of "Soon"

Go back to our order. The user sees "Order Confirmed" instantly. The invoice appears seconds later. Loyalty points a moment after that. The search index catches up next. For a brief window, different parts of the system disagree about the state of the world. On purpose.

This is eventual consistency — the same idea we met with stale caches in Part 6 and replicated databases in Blog 5, now applied to workflows: the system is temporarily out of sync, and converges to correct.

The architect's question is never "is eventual consistency bad?" It's: for this workflow, how long can "eventually" be, and what does the user see in the meantime? Points appearing after 30 seconds? Nobody notices. The order itself flickering between placed and not-placed? Unacceptable — which is exactly why the order write was synchronous and only the follow-up work was async.

That line — what must be now and what may be soon — is one of the most important lines you will ever draw in a design.

What Not to Put in a Messaging System

By now, async sounds like the answer to everything. It isn't. Knowing when not to reach for a queue is half the skill — same as Part 6's "what not to cache."

Simple synchronous reads. "Get user profile" does not need a queue. The user is standing right there, asking. Answer them. A queue adds latency, machinery, and confusion to something request-response already does perfectly.

Workflows that need an immediate user response. Payment authorization at checkout. Login. "Is this seat still available?" The user is waiting for the outcome. Making them poll for a result that could have been returned directly is architecture cosplay.

Operations where delayed consistency is unacceptable. Part 6's rule was: never cache the confirmation check. The async twin: never make the confirmation itself eventual. The final booking write, the balance check before withdrawal — these are now operations. Ask, confirm, respond.

Messages without clear ownership. If nobody owns a queue, nobody drains its DLQ, nobody notices its lag, and nobody answers for its failures. An unowned queue is an unowned outage, scheduled for later.

Events with unclear schema or versioning. The producer renames a field. Three consumers silently start parsing garbage, at 2 AM, in another team's service. An event schema is a contract. Write it down, version it, and treat changes like the API changes they are.

Business processes where no one owns failure handling. "We publish the event" is not a process. Who acts when it fails five times? If the answer is a shrug, the process isn't designed yet — it's just optimistic.

Queues used to hide slow code. Part 6's mask, again. A 40-second job pushed onto a queue is still a 40-second job, now with worse visibility and a growing backlog. Sometimes async is the right call for genuinely heavy work. But if you never ask why it's slow, you haven't fixed the problem — you've scheduled its return.

Messaging Makes Systems Scalable — But Harder to Understand

Time for the honest bill. Everything in this article trades one thing for another. What you gain in decoupling and resilience, you pay for in understandability.

Debugging changes shape. In a synchronous world, an error gives you a stack trace — one timeline, one story. In an async world, the failure is a distributed timeline: the order API succeeded at 14:02, the invoice consumer failed at 14:07, the retry failed at 14:15, the DLQ alert fired at 14:31. Four services, four log files, one bug.

Failures become delayed. The user got "success" and left. The failure happened later, elsewhere. Nobody is on the page anymore to see an error message. If your system doesn't tell someone, nobody knows.

Bugs become partial. The email went out. The invoice didn't. The loyalty points doubled. Async bugs are rarely "it's down" — they're "it half-happened."

The survival kit: correlation IDs (stamp one ID on the original request and pass it through every message it spawns — suddenly four log files become one searchable story), the new vital signs (queue depth, consumer lag, DLQ size, processing time, retry rate — these are to async systems what CPU and memory are to servers), and alerts on the DLQ (a non-empty DLQ is a customer-facing bug in a waiting room).

This is exactly why observability stops being optional the moment messaging enters your architecture. We'll build that discipline properly in Part 12: Observability — Designing Systems That Tell You When They Break.

Applying C.R.E.D to Messaging

The same framework since Part 3, one more time.

Clarify — For each piece of work: does it need to happen now, or soon? Does the user need the result to proceed?

Requirements — Per workflow: which delivery guarantee? What ordering, and per what key? How long may "eventually" be?

Estimate — Message rate at peak. Burst size. Consumer throughput. How fast does lag grow during a spike, and how fast can you drain it?

Design — Queue or stream. Command or event. Retry policy. DLQ and its owner. Idempotency strategy. The consistency window the user will actually experience.

If you can fill those blanks, you're not "adding Kafka." You're designing coordination. And that's the real shift: a good architect does not ask only, "Should these two services call each other?" A good architect asks, "Does this work need to happen now, later, once, many times, in order, or at scale?"

From My Journey: An Architectural Lesson

In many systems I've seen, the story starts the same way: direct API calls everywhere. And honestly, that's the right start — direct calls are easy to understand, easy to trace, easy to debug. One request, one response, one log to read.

Then the workflows begin to accumulate. One user action starts triggering notifications, reports, audit logs, emails, analytics, search indexing, billing updates, and external integrations. What was one call quietly becomes nine.

The challenge, we assumed at first, was performance — responses were getting slower. But the real problem was hiding one level deeper: coupling. When every downstream operation must finish before the user gets a response, the system doesn't just become slow. It becomes fragile — any downstream failure breaks the main flow — and it becomes difficult to evolve, because every new requirement means touching the critical path.

Messaging changed that. The main flow shrank back to its essentials, and everything else became work that happened soon. But it wasn't free. We inherited a new set of responsibilities: retries, duplicates, ordering, delayed failures, and the discipline of actually watching queue lag and dead letters.

That experience left me with the lesson this article is built on: messaging is not just a scalability tool. It is a coordination model. A queue is not a dustbin for slow work. It is a promise that someone will own the work later.

Key Takeaways

• Synchronous chains couple latency and availability — every downstream service can slow or break the main flow.
• The producer → broker → consumer model decouples when work is requested from when it is done.
• Commands say "do this" to one owner; events say "this happened" to whoever cares.
• Queues distribute tasks — consumers compete and messages disappear. Streams remember events — consumers are independent and can replay history.
• Ordering is a per-key decision, not a global one.
• Retries need limits, and failing messages need a dead-letter queue with a named owner.
• Delivery guarantees are business decisions: at-least-once delivery plus idempotent processing beats chasing perfect exactly-once.
• Watch consumer lag — a queue that only grows is a crash on a payment plan.
• Async buys scale and resilience at the cost of understandability; correlation IDs and observability are the price of admission.

Interview Lens

Messaging questions are rarely asked directly. They hide inside design questions.

The classic probe: "A user uploads a video. What happens next?" Weak answer: a chain of synchronous calls. Strong answer: acknowledge fast, then name the async pipeline and the guarantees.

The template, parallel to Part 6's caching sentence: "I'd publish a VideoUploaded event; transcoding workers consume with at-least-once delivery and idempotent processing, exponential backoff on retries, a DLQ with alerts, and per-video ordering via the video ID as the key." Guarantee. Retry. DLQ. Idempotency. Ordering key. Five blanks, one sentence.

Expect these follow-ups: "Queue or Kafka?" — answer with the distinction, not the brand: task to be done once → queue; event history multiple consumers may need or replay → stream. "How do you handle duplicates?" — idempotency keys in business logic; saying "the broker guarantees exactly-once" is how this question is failed. "What if the consumer is down for an hour?" — messages wait, lag grows, consumers drain the backlog on recovery — then mention backpressure limits. "How do you keep events in order?" — per-key ordering via partitioning; global ordering is almost never worth its cost.

The differentiator: most candidates bolt messaging on for scale. Strong candidates use it to draw the now/soon line — "the order write is synchronous; everything downstream is eventual, with these windows." That sentence signals architecture maturity instantly.

Real-World Engineering Lens

Start with one boring queue. Background jobs — emails, PDFs, image processing — on RabbitMQ or SQS. Adopt streams when multiple consumers genuinely need the same events, or replay starts mattering.

Decide the guarantee per workflow, in writing. One line in the design doc: "Invoice generation: at-least-once, idempotent on order ID, DLQ owned by billing team." Future-you, again, will be grateful.

Monitor lag and the DLQ from day one. An unwatched DLQ is where customer bugs go to age. Alert on non-empty.

Version your events from the first schema. Adding optional fields is easy; renaming is a breaking API change wearing casual clothes.

Stamp correlation IDs before you need them. The day you're reconstructing a distributed timeline at 2 AM is the wrong day to start.

Test the duplicate. Replay a message in staging and watch what breaks. Whatever double-fires is your idempotency to-do list.

What's Next

Our events now carry facts like "video uploaded" and "invoice generated." But we've also spent this entire article deciding, deliberately, that some things can be true "soon" instead of "now."

Bonus I — Replication Lag and Eventual Consistency in Practice. The next article grounds that idea in the specific case of database replicas: why copies fall behind, what problems that lag actually causes for real users, and the patterns — read-your-writes, monotonic reads, and more — that keep "eventually" from becoming "whenever."

After that, Blog 8 — Storage Systems picks up a question our events have been quietly begging: where do the actual bytes — the video, the invoice PDF, the retained event log itself — actually live? Where we'll find out why "just put it in S3" is doing a lot more heavy lifting than most engineers realize.

Question for Readers

Look at your current system and find one workflow where the user is kept waiting for something they don't actually need to wait for.

What would it take to make it asynchronous? And more importantly — which new failure would you then need to own: the retry, the duplicate, or the lag?