Chat Application

Two people are typing to each other. Between them: two flaky radios, four devices, a dozen servers — and one shared belief about what was said, in what order.

The system's job is protecting that belief.

Part 13 designed a system whose data stood still. Part 14 designed one whose data moved one way — service to user, fire and track. This one removes every remaining comfort at once. Both ends are humans, both are waiting. The connection stays open — a direct challenge to Part 9's stateless serenity. "Eventually" is measured in milliseconds. And Part 14's teaser question is now due: what does "delivered" mean when the recipient's phone just went through a tunnel?

Same template, third lap. Last one of Section 2 — let's make it count.

Why the Chat Application Is the Perfect Final Mini-System

It breaks the request-response worldview. Everything since Part 4 assumed a request arrives, a response leaves, the connection ends. Chat holds millions of connections open, idle, and waiting — a completely different resource economy.

It makes distributed state user-visible. Ordering, delivery, presence, read status — every abstract concept from Parts 7 and Bonus I becomes a pixel a user stares at. When eventual consistency misbehaves here, it isn't a stale dashboard; it's "why does my chat show messages out of order?"

It integrates the previous mini-system. Offline users get pinged through — yes — Part 14's notification service. Section 2's systems aren't three exercises; they're becoming an estate.

And it's the honest test of "delivered." Most engineers discover in this problem that they've never actually defined the word.

A good architect does not ask only, "Can two people exchange messages?" A good architect asks, "What does delivered mean, in what order must this arrive, and what happens when the connection goes quiet?"

Step 1 — Clarify: The Questions Before the Boxes

"Build a chat app" spans everything from a two-person messenger to a million-member channel. The separating questions: 1:1 only, or groups — what's the group size ceiling? Receipts and typing indicators — sent/delivered/read ticks? Multi-device — phone plus laptop, history synced, read-state shared? Media — photos, videos, documents? Presence — online/offline/last-seen? End-to-end encryption?

Our assumptions, stated:

Conversations:    1:1 and groups, capped at 256 members
Receipts:         Yes — sent / delivered / read, per recipient
Typing:           Yes — best-effort
Multi-device:     Yes — full history sync, shared read state
Media:            Yes — via reference (the bytes go elsewhere)
Presence:         Yes — approximate is acceptable
E2E encryption:   OUT of this design — with the reasoning documented

That 256 cap is load-bearing: it's the line between "group chat" and "broadcast channel" — and channels are a different system (Part 16 is waiting for exactly this). Capping it is how we keep this design honest.

Step 2 — Requirements: The Building Code

Functional: send/receive messages in real time; per-conversation history; delivery and read receipts; typing indicators; presence; offline delivery on reconnect; multi-device sync; media sharing by reference.

Non-functional — and notice the guarantees vary by message type, exactly as Part 14 taught:

Chat messages:
  Online-to-online latency p99    < 500 ms
  Ordering                        per conversation — strict; global — never
  Loss                            ZERO once the sender sees "sent"
  Duplicates                      never shown twice (client-visible effectively-once)
  Offline delivery                complete, on reconnect, in order

Typing indicators & presence:
  Delivery                        best-effort — at-most-once is CORRECT here
  Freshness                       seconds; approximate by design

History:
  Retention                       1 year hot, then archive (Part 8)
  Multi-device                    all devices converge to identical history

Read that middle block twice. Part 7 introduced at-most-once as the guarantee for "low-value signals" — this is its natural habitat. A dropped typing indicator is invisible. A retried one is a ghost typing forever. Choosing the weakest guarantee on purpose is as much a design act as choosing the strongest.

Step 3 — Estimate: The Numbers

Bonus A's method, plus one new unit this problem forces into the vocabulary:

Assume: 10M DAU, ~40 messages sent per user per day

Messages:      400M/day ÷ ~100K sec ≈ 4,000/sec average
               × 5 peak            ≈ 20,000/sec at peak

Group fan-out: avg 1 send → ~3 deliveries (mix of 1:1 and groups)
               → delivery events ≈ 60,000/sec at peak

Storage:       400M × ~1 KB ≈ 400 GB/day ≈ 12 TB/month ≈ 144 TB/year
               → the biggest data in Section 2 by two orders of magnitude
               (Part 13: ½ TB total. Part 14: 4.5 TB. This: 144 TB/year.)
               → sharding is NOT optional here; archive tiers from day one

THE NEW UNIT — concurrent connections:
               ~20% of DAU online at peak ≈ 2M open WebSockets
               ÷ ~150K connections per gateway node (memory-bound, assumed)
               ≈ 15–20 connection gateway nodes, before headroom

The estimation verdict: Part 13 was reads, Part 14 was writes-in-bursts — this is connections and fan-out. The scarce resources are open sockets and delivery events, not QPS. And at 144 TB/year, the storage design graduates from afterthought to first-class citizen: sharded from birth (Bonus G already told us the key — hold on).

Step 4 — Design: The High-Level Architecture

The defining split, stated before the diagram: the connections are stateful; everything else stays stateless. Part 9's discipline isn't abandoned — it's fenced. One layer is allowed to hold state (open sockets), kept as thin as possible, and everything behind it remains cattle.

Devices (phone, laptop — each its own connection)
    │  WebSocket, long-lived, heartbeat every ~25s
    ▼
CONNECTION GATEWAYS  (the stateful fence — sockets live here)
    │        ▲
    │        │  Session Registry (Redis): user+device → gateway node
    ▼        │  (heartbeat-refreshed, TTL'd — derived, evictable: Bonus H)
CHAT SERVICE (stateless — Part 9 intact)
    │
    ├─▶ MESSAGE STORE  ← source of truth; sharded by conversation_id
    │        │              assigns per-conversation sequence numbers
    │        ▼
    ├─▶ Fan-out via broker (Part 7): route to each recipient device
    │        ├─ device ONLINE  → its gateway → push down the socket
    │        └─ device OFFLINE → Part 14's notification service
    │
    └─▶ Media: upload direct to object storage via signed URL (Parts 8, 11);
        the message carries the REFERENCE — Part 7's oldest rule, live

Three inherited decisions worth naming. The store map (Bonus H): message store is truth; session registry, presence, unread counters are derived and evictable; media bytes are object storage with the message holding the key. The shard key (Bonus G): conversation_id, hashed — all of a conversation's messages co-locate (ordering stays cheap), conversations distribute evenly, and the 256-member cap bounds the worst hot key. The integration (Part 14): offline delivery doesn't reinvent push, it publishes to the notification service's transactional lane. The estate compounds.

Deep Dive 1: The Connection Layer — Holding Two Million Open Sockets

Why WebSockets at All

The request-response alternative is polling: every client asks "anything new?" every couple of seconds. Run the arithmetic (Bonus A reflex): 10M clients polling every 2 seconds equals 5 million QPS of mostly-empty responses. A WebSocket inverts it: one handshake, then a quiet open pipe — 2M idle sockets cost memory, not requests, and messages push down them the instant they exist.

Polling pays per question. Sockets pay per connection. At chat's read-to-write shape, connections win by orders of magnitude.

The Gateway's Job Description

Each gateway node holds ~150K sockets and does deliberately little: authenticate the handshake (Part 11 — a token, verified before the socket is accepted), register user+device → me in the session registry, heartbeat, and shuttle frames. No business logic lives here — the thinner the stateful layer, the less it hurts.

Heartbeats: How the Tunnel Is Detected

TCP will happily believe a vanished phone is "connected" for a long time. So both sides ping: client and server exchange heartbeats every ~25 seconds. Miss a few → the connection is declared dead, the registry entry expires (its TTL was always waiting for this), and deliveries for that device flip to the offline path.

Presence falls out of the same mechanism — but that's Deep Dive 3.

Reconnection: The Stampede You Must Plan For

Here's the scenario that separates chat designs: a regional network blip drops 500,000 connections in one second. Every client immediately tries to reconnect.

Unmanaged, that's a self-inflicted DDoS — Bonus E's retry storm at connection scale. The defenses, all pre-met: jittered exponential backoff in the client (the reconnect wave spreads over a minute instead of a second); the handshake path is the cheapest path (auth-check + registry-write only, no history sync in the handshake — that comes after, paced); and gateways are behind least-connections balancing (Bonus E) — reconnecting users spread across the fleet instead of refilling the recovering node.

Deploys Disconnect Users — Design for It

Part 9 said restarts wipe in-process state; here the in-process state is people's conversations' pipes. Rolling deploys drain gateways (Bonus E's connection draining, applied to sockets): stop accepting new connections, nudge existing clients to reconnect elsewhere, then restart. Clients treat it as an ordinary blip. Deploys become invisible — because the reconnect machinery above was built for worse.

Deep Dive 2: Ordering, Delivery States, and the Tunnel Answer

Ordering: Per Conversation, Assigned by the Server

Part 7 taught the law: global ordering is expensive and unnecessary; per-key ordering is the practical answer. Chat's key is the conversation.

The message store assigns each message a per-conversation sequence number at write time. Two rules with teeth: the server assigns it, never the client's clock — device clocks lie (drift, timezone games, users changing settings), sequence numbers don't. And cross-conversation order is explicitly unpromised — your work chat and your family group may interleave differently on your two devices. Nobody has ever noticed. Within a conversation, never.

Sharding by conversation_id makes this cheap: one conversation, one shard, one counter. (A single global counter would be Bonus G's hotspot; per-conversation counters are millions of tiny cold ones.)

The Three Ticks: A State Machine, Not Decoration

SENT       server durably stored the message and acked the sender
DELIVERED  the recipient's DEVICE acknowledged receiving it
READ       the recipient actually viewed it

And here is the tunnel question, answered precisely: "delivered" is the recipient device's acknowledgment, never the server writing to a socket. The gateway can push a frame into a connection whose phone entered a tunnel 4 seconds ago; TCP hasn't noticed; the frame is gone. If "delivered" meant "we wrote to the socket," the tick would lie.

So the protocol is Part 7 all the way down:

Server → device:  message (seq 42)
Device → server:  ACK 42            ← only NOW is it "delivered"

No ACK in time →  redeliver on next connection
Device sees seq 42 twice →  client-side dedup by message ID
                             (idempotency at the edge — Part 14's
                              lesson, running on a phone)

At-least-once delivery plus idempotent receiver equals a tick that tells the truth. The guarantee negotiation from Part 14, miniaturized into every chat bubble.

Multi-Device: Every Device Is Its Own Consumer

The phone and the laptop don't share a connection — they share a history. The mechanism is one you already know from Bonus I: each device keeps a cursor — "last sequence I have, per conversation."

Laptop reconnects after 2 hours:
  "conversation 881: I have up to seq 4,507"
  → server streams 4,508…4,631, in order
  → cursor advances

Offline delivery and multi-device sync turn out to be the same feature: cursor catch-up. A device that was in a tunnel and a laptop that was closed overnight are indistinguishable to the protocol — both are consumers behind on their offsets, catching up. (Read receipts fan back the same way: you read on the phone; the laptop's badge clears because the read-state change is just another event flowing to every device cursor.)

Deep Dive 3: Presence — Honest Lies with Timeouts

Presence looks like a boolean. It's actually a derivative:

online  =  "a heartbeat from this user arrived < 30 seconds ago"

That's it. Presence is approximate by construction — the phone in the tunnel shows "online" for up to 30 seconds after it's gone, and no design fixes that without heartbeating so aggressively you melt the batteries. State the staleness; stop apologizing for it. (Bonus F's question, answered: wrong for anyone watching, for ≤30 seconds, self-healing.)

Two design rules keep presence cheap. It's derived and evictable (Bonus H) — presence lives in the registry's TTL'd entries; lose the store, thirty seconds of heartbeats rebuild it perfectly. Anything that can be rebuilt from heartbeats must never be treated as truth. And fan-out on demand, not on change — the naive design publishes "user X came online" to all 500 of X's contacts, and 2M users churning connections becomes a presence storm dwarfing the actual chat traffic. The honest design: push presence only for conversations currently on screen; pull it for contact lists when opened. Presence for someone you aren't looking at is a wasted delivery.

And typing indicators, the most disposable data in the system, are fire-and-forget frames between online parties. No store, no retry, no receipt. At-most-once, proudly.

Failure Analysis: Where This Design Bends and Breaks

A gateway node dies. 150K sockets vanish at once. Clients notice within a heartbeat interval and reconnect — jittered — across the surviving fleet; registry entries expire on their own TTLs; in-flight-but-unACKed messages redeliver on reconnection (the state machine never trusted the socket anyway). Cost: a sub-minute blip for 7% of online users. The design's whole posture is that this is Tuesday.

The message store write fails. The sender never gets the "sent" tick — the client retries with the same client-generated message ID, and the server dedupes (Part 14's claim pattern, upstream edition). The user sees a clock icon, then one tick. Never two copies.

The registry is stale — a message routes to a gateway whose socket just died. The push fails or times out unACKed, and the delivery flips to the offline path (store plus Part 14 ping). The fallback path is the correctness path; the socket was only ever the fast lane.

A 256-member group gets chatty. One send → 256 delivery events × device counts. The fan-out worker pool absorbs it (Part 7); the cap bounds it. This is also exactly where the design says its own limit out loud: a 100,000-member "group" isn't a group — it's a broadcast feed, and feeds have different physics. Part 16 exists because of this sentence.

A regional blip reconnects 500K clients. Deep Dive 1's stampede plan executes: jitter spreads the wave, cheap handshakes survive it, history sync trickles afterward via cursors. The metric that proves it worked: reconnect-success p99, not connection count.

Read receipts race across devices. You read on the phone while the laptop is mid-sync. Both converge — receipts are events with sequence positions, and cursors don't skip. Out-of-order display is prevented by the same per-conversation ordering that protects messages.

Every failure: named, bounded, answered by machinery chosen in advance — with one honest asterisk: the 30-second presence lie, accepted in writing.

Security and Abuse: The Part Everyone Skips

A chat system carries people's most private text through your infrastructure. The defensive design (Part 11 throughout): authenticate the handshake, and keep authenticating (the token is verified before the socket is accepted, but a WebSocket can outlive its token — long-lived connections need a policy for token expiry mid-connection: re-authenticate over the open socket, or drop-and-reconnect on expiry; choose one, never let a connection's age become an authorization bypass). Authorize every message, not every connection ("connected" is not "member of this conversation" — the chat service checks conversation membership on each send, the socket is a pipe, not a permission; Part 11's rule: the token is evidence, the connection is just plumbing). Rate-limit at the message level, per user, per conversation (an unthrottled sender is a spam cannon with a captive audience). Media through signed URLs only (Parts 8, 11) — upload and download both, the chat pipeline never carries bytes, and links expire. Metadata is sensitive too — who talks to whom, when, how often; logs and analytics carry conversation existence, never content, and even the existence data honors Part 12's PII discipline. And block and report as first-class flows — blocking silently severs delivery without notifying the blocked party; reports snapshot evidence with audit trails (Parts 11–12).

The Observability Plan

Part 12's table, applied — with the connection layer earning new rows:

SLO:          99.9% of online-to-online messages delivered < 500 ms
Connections:  concurrent sockets per node, connect/disconnect rate,
              reconnect-success p99, heartbeat miss rate
Delivery:     end-to-end p99 (sent→delivered ACK), redelivery rate,
              client-dedup hit rate  ← the smoke detector, again
Ordering:     out-of-sequence detections (should be ~zero; alert if not)
Store:        write p99 per shard, hot-conversation top-K (Bonus G),
              cursor lag distribution across devices
Fan-out:      delivery events/sec, worker lag, offline-path handoffs to Part 14
Presence:     registry TTL expiries, presence-accuracy sampling
Business:     messages/day, DAU messaging, median conversation size
Synthetic:    two robot users per region exchanging and ACKing messages
              every minute — the tunnel test, automated

The reconnect-success p99 deserves the highlight: it's the metric that turns "gateway died" from an incident into a footnote — or reveals that your stampede plan is fiction.

What Interviewers Actually Probe

"Polling or WebSockets — why?" — run the arithmetic out loud: 5M QPS of empty polls vs. 2M idle sockets; paying per question vs. per connection; the numbers are the answer. "What does 'delivered' mean, exactly?" — the tunnel question; device ACK, never socket write, and the redelivery plus client-dedup machinery behind the tick; this single answer separates candidates more than any other in this problem. "How do you order messages?" — per-conversation sequence, server-assigned, client clocks never trusted, global order explicitly unpromised; bonus points for naming the shard-key alignment. "A group has 100,000 members." — the trap with a cap: "then it's not a group, it's a broadcast feed — different fan-out physics, different design" (and in this series, a different article). "What happens on deploy?" — drain the sockets, jittered client reconnects, cheap handshakes; candidates who volunteer that deploys are connection events think in production. "Phone and laptop — how do they stay in sync?" — per-device cursors; offline delivery and multi-device sync as the same catch-up mechanism. "Half a million clients reconnect at once." — jittered backoff, handshake-only reconnect path, least-connections spreading, the stampede plan, recited calmly.

What We Deliberately Didn't Build

End-to-end encryption — the honest omission, with the honest reason: E2E fundamentally changes the server's role from participant to blind router. Server-side search, multi-device history sync, and abuse moderation all get dramatically harder, because the server can no longer read what it stores. It's a real product direction with real trade-offs (Bonus C: a business decision with heavy muscle requirements), not a checkbox to retrofit. Broadcast channels and mega-groups — the 256 cap was the fence; feeds and fan-out at scale are Part 16's opening problem. Voice and video calling — signaling could ride this design; media transport is a different discipline entirely. Message editing and deletion propagation — real features, straightforward extensions of the event-and-cursor model; scope, not mystery. Federation (chatting across providers) — a protocol standardization problem wearing a system design costume.

Common Mistakes Engineers Make with This Problem

Trusting client timestamps for ordering — device clocks lie, sequence numbers don't. "Delivered" equals wrote to the socket — the tick that lies, the tunnel finds it every time. No client-side dedup — at-least-once redelivery with no idempotent receiver means every reconnect shows double messages. Business logic in the gateway — the stateful layer must stay thin, every feature added to it multiplies the cost of node failure. Unjittered reconnects — one regional blip becomes a self-inflicted DDoS. Presence pushed to every contact on every change — the presence storm that outweighs the chat traffic. Treating the registry as truth — it's a TTL'd cache of "who's where," the store-and-notify path is the correctness path. A global message counter — Bonus G's hotspot, installed at the highest-write point in the system. Sharding by user instead of conversation — a conversation's messages scattered across shards makes ordering, the core guarantee, expensive. Ignoring token expiry on long-lived connections — the socket that outlives its authorization. Retrying typing indicators — the ghost that types forever, at-most-once existed for a reason. No stampede plan — the design that works until the first regional blip, which is to say, until Thursday.

From My Journey: An Architectural Lesson

Chat has a reputation as an approachable problem — and the first version reinforces it. Two clients, a server relaying strings between them, messages appearing instantly. An afternoon's work, and it feels real-time.

Then the real questions arrive, each one innocent: what if the recipient's app was closed? What order do messages appear in when two people type simultaneously? Why does the phone show a message the laptop doesn't? What should the tick mean, and when is it allowed to appear?

The realisation that reframes the problem: sending messages was never the feature. Agreement is the feature. Two people — across four devices, flaky radios, and a dozen servers — must end up believing the same conversation happened, in the same order, with the same understanding of what was seen. Every chat UI element that looks like decoration — ticks, ordering, presence, unread badges — turns out to be a distributed-state problem wearing a pixel.

And the discipline that falls out is guarantee literacy, again: strict per-conversation ordering here, best-effort typing indicators there, a presence lie with a documented timeout — each guarantee priced and chosen, none inherited by accident.

The lesson that stuck: a chat message is not delivered when it arrives. It is delivered when both sides agree it arrived.

Key Takeaways

• Chat's scarce resources are open connections and fan-out events — a different economy from Part 13's reads and Part 14's bursts.
• Fence the state: gateways hold sockets and nothing else; every layer behind them stays stateless (Part 9, preserved under pressure).
• "Delivered" means the recipient device ACKed — at-least-once redelivery plus client-side dedup makes the tick honest.
• Ordering is per-conversation, server-assigned sequence numbers; client clocks are never consulted; global order is unpromised on purpose.
• Multi-device sync and offline delivery are one mechanism: per-device cursors catching up — Bonus I's offsets, running on phones.
• Presence is a derivative with a documented staleness window — derived, evictable, pushed only for what's on screen.
• Typing indicators are at-most-once, proudly — choosing the weakest guarantee is a design act.
• Shard by conversation_id; cap groups at the fence where chat ends and feeds begin.
• Reconnect stampedes are the failure to plan for: jittered backoff, cheap handshakes, drained deploys.
• Offline users route through Part 14's notification service — Section 2's systems compound into an estate.

Interview Lens

Third lap, same winning shape, with this problem's signature move up front: define "delivered" before anyone asks. The moment you say "delivered means the device acknowledged, so the protocol is redelivery plus client dedup," you've answered the tunnel question, demonstrated Part 7's guarantee literacy, and marked yourself as someone who's thought past the happy path — all in one sentence.

Then the numbers declare the problem ("connections and fan-out — my budget goes to the gateway layer, ordering, and the stampede plan"), the state gets fenced, and the failure walk arrives uninvited: gateway death, regional reconnect, the 100K "group."

The meta-skill this problem tests: knowing which guarantees each feature deserves. Messages get the strong ones. Typing gets the weakest. Presence gets an honest lie. The candidate who prices guarantees per feature, instead of maximizing everywhere, is the one the debrief remembers.

Real-World Engineering Lens

Find your long-lived connections and audit their auth. Any WebSocket or streaming connection in your estate: what happens when its token expires mid-flight? "Nothing" is a finding.

Check your real-time features for the dedup gap. If anything pushes over sockets with retries and no client-side idempotency, your users have seen doubles — they just haven't filed it yet.

Run the stampede drill. Kill a gateway (or its equivalent) in staging and watch the reconnect curve. Jitter that exists only in a design doc protects nothing.

Put reconnect-success p99 on the dashboard next to connection count. The count says how many; the p99 says whether recovery actually works.

Steal the cursor model for any multi-device or offline-capable feature — it's the cheapest correct sync mechanism there is.

What's Next

Stop and look at what Section 2 built. Three systems: one that reads (Part 13), one that delivers (Part 14), one that converses (Part 15) — each walked through the same discipline: clarify, estimate, design, break it on paper, secure it, instrument it. The template held. The building blocks composed. The estate compounds.

Section 2 is complete. Section 3 raises the stakes from single systems to categories, and it opens with the fence we just built.

Blog 16 — Social Networks and Feeds: Fan-out, Ranking, and Real-time Delivery. The group chat that grew to a million members. The celebrity problem, finally faced head-on (Bonus G has been waiting since its first mention). One post, five million followers — where does the write go, and who pays for the read?

Question for Readers

Open any chat app on your phone and watch the ticks on your next message.

Now answer, for your own product's real-time features: what exactly does your "delivered" mean — a socket write, or an acknowledgment?

If you're not sure, somewhere a tunnel is already proving it's the wrong one.