Collaboration Tools

Every system so far had one writer per fact. One order. One inventory count. One source of truth, and everyone else waiting their turn.

Now ten people type into the same sentence at the same instant, and every one of them is a writer.

Part 18 held money correct with a single writer as the source of truth per record. There was always one authoritative copy of the order, the inventory, the payment, and the whole design was protecting that single writer from races.

This article removes the single writer entirely.

In a collaborative editor, there is no "the" version being edited. There are ten local copies, diverging every keystroke, that must somehow converge to an identical result, with no one taking turns, no lock, and no waiting. A promise Bonus I made back in the replication article is finally due: "two desks both edit their copies? That's a conflict, a different beast, waiting for us in Part 19."

Welcome to the beast. Same walkthrough template, the hardest consistency problem yet.

Why Collaborative Editing Breaks Normal Thinking

Concurrent writes are the normal case, not the failure. Every system before this treated simultaneous writes to the same thing as a race to prevent (Part 18) or an ordering to enforce (Part 15). Here, ten people editing one paragraph at once isn't an edge case to guard against, it's Tuesday. The architecture is built around concurrency, not defended from it.

"Last write wins" is a data-loss bug, not a strategy. The reflex answer to a conflict, keep the latest write, silently destroys someone's work. In commerce that's a race; in a document it's "my paragraph vanished while I was typing it." The entire discipline exists because the easy answer is unacceptable.

Correctness means convergence, not a single truth. Part 18's correctness was "exactly once." Here it's weaker and stranger: every replica, after all edits are exchanged in any order, ends up identical, and ideally, matching what the humans intended. There's no server-held ground truth to check against mid-edit; the truth is the agreement the replicas reach.

And latency can't hide behind a spinner. A user typing expects their own characters to appear instantly, you cannot round-trip to a server per keystroke. So every client edits a local copy first and reconciles after, which means divergence is designed in. This is the opposite of Part 18's "fail closed, confirm before proceeding."

A good architect does not ask only, "How do we let people edit together?" A good architect asks, "What does 'correct' even mean when there's no single writer left to protect?"

Step 1 — Clarify: The Questions Before the Boxes

"Build a collaborative editor" spans a shared text doc, a spreadsheet, a whiteboard, and a design canvas — related but not identical. The separating questions: what's being edited — plain text, rich text, a spreadsheet grid, freeform shapes? The data type changes the conflict math entirely. How many concurrent editors — two (a shared note) or hundreds (a live doc in a big meeting)? Real-time or async — live cursors and instant sync, or occasional saves that merge? Offline editing — can someone edit on a plane and reconcile on landing? History and versioning — full undo, named versions, blame-per-character? Presence — live cursors, selections, who's-here?

Our assumptions, stated (Bonus A's rule):

Content:       Rich text document (the canonical hard case)
Editors:       Up to ~50 concurrent per document
Sync:          Real-time — edits appear in ~100ms, cursors live
Offline:       Yes — edit offline, converge on reconnect (the hardest requirement)
History:       Version history + undo/redo
Presence:      Live cursors and selections (Part 15's machinery, reused)

That offline answer is the one that forces the deepest machinery. Real-time-only collaboration has easier options; "edit for an hour on a plane, then converge cleanly" is what rules out the shortcuts and demands OT or CRDTs.

Step 2 — Requirements: The Building Code

Functional: multiple users edit one document simultaneously; edits sync in real time; concurrent edits merge without losing work; live cursors/presence; version history and undo; offline edits reconcile on reconnect.

Non-functional — and notice how different this shape is from Part 18 (Bonus B, per data type):

CONVERGENCE (the defining requirement):
  Guarantee              all replicas reach an IDENTICAL state after
                         all operations are delivered, in ANY order
  Intent preservation    the merged result should reflect what users MEANT,
                         not just a mechanically valid string

LOCAL LATENCY:
  Own-keystroke feedback  instant (< 16ms) — never wait for the server
  Remote-edit visibility  < ~100ms when online

CONSISTENCY MODEL:
  Strong consistency      explicitly REJECTED — it would require locking/turns
  Eventual + convergent   the goal: temporarily divergent, guaranteed to converge

DURABILITY & HISTORY:
  No lost edits           an acknowledged edit is never dropped
  Reconstructable history full version timeline + undo

OFFLINE:
  Divergence window       may be long (hours); convergence on reconnect must be clean

Read the consistency block twice. This is the first system in the series where strong consistency is the wrong answer on purpose. Part 18 fought for strong consistency where money lived; here, demanding it would mean locking the document so only one person types at a time, which defeats the entire product. Bonus F's dial, turned deliberately to the eventual-but-convergent setting.

Step 3 — Estimate: The Numbers That Shape the Design

Bonus A's method, and the surprising part is where the load actually is:

Assume: a document with 50 active editors during a busy meeting

EDIT OPERATIONS (the real-time firehose)
  Active typing: ~5 keystrokes/sec per engaged user
  Say 10 actively typing at once → ~50 ops/sec PER DOCUMENT
  Each op is TINY (insert char, delete range) but must fan out
  to all 50 editors → ~2,500 op-deliveries/sec per hot document

FAN-OUT (Part 15's shape, per document)
  Each op broadcasts to every other connected editor
  → this is a small-scale, high-frequency fan-out (Blog 16's cousin)

CONNECTIONS (Part 15's economy, reused)
  50 editors = 50 open WebSockets per active doc
  Millions of docs, most idle → connection management dominates

STORAGE
  The document is small (~KBs of text)
  But the OPERATION HISTORY can dwarf it — every keystroke, versioned
  → history compaction/snapshotting becomes a real design concern

The verdict flips intuition: the document is tiny; the operation stream is the system. The load isn't data volume, it's a high-frequency flood of micro-operations that must be ordered, merged, fanned out, and compacted. This is Part 15's connection-and-fan-out economy meeting Part 18's correctness stakes, a genuinely new combination.

The Core Problem: Why Concurrent Edits Conflict

Before any solution, feel the problem precisely. Two users share the text "cat". Simultaneously: User A inserts "s" at position 3 → intends "cats". User B inserts "!" at position 3 → intends "cat!".

Both edits say "insert at position 3." Both were computed against the same original "cat". Now they must combine, and naive application destroys intent:

Apply A then B literally:  "cat" → "cats" → insert "!" at 3 → "cat!s"  ?
Apply B then A literally:  "cat" → "cat!" → insert "s" at 3 → "cats!"  ?

Two users, two machines, TWO DIFFERENT RESULTS. Divergence.

There it is: the same operations, applied in different orders, produce different documents. That's the whole problem. Position 3 meant something different the instant the other edit landed — B's "!" shifted where A's "s" should go, and vice versa. An edit's coordinates are only valid against the state it was written for; once a concurrent edit changes that state, the coordinates lie.

"Last write wins" would resolve this by throwing one edit away. Correct convergence resolves it by making every replica agree on the same result and keeping both users' intent. Two families of algorithms do this.

Deep Dive 1: Operational Transformation (OT)

The older, battle-tested approach — the engine behind the most famous collaborative editors.

The Idea: Transform Operations Against Each Other

OT keeps the operations but rewrites their coordinates as they pass each other, so intent survives regardless of order:

A = insert "s" at 3   B = insert "!" at 3   (both vs "cat")

When B arrives at a replica that already applied A:
   TRANSFORM B against A → B' = insert "!" at 4
   (A already added a char before position 3's neighborhood,
    so B's target shifts right)

Result everywhere:  "cat" + A + B' = "cats!"    ← identical on every replica

The transformation function is the heart of OT: given two concurrent operations, it produces adjusted versions such that applying them in either order yields the same convergent result. Do it for every pair, in every order, and all replicas converge.

What OT Costs

A central server usually orders the operations — most OT systems route edits through a server that assigns a canonical order and transforms against it, which simplifies the math but makes the server a coordination point (and a scaling concern per document). The transformation functions are notoriously hard to get right — every operation type against every other, in every order, must be proven convergent; rich text (bold, links, structure) multiplies the cases brutally, this is where OT earns its reputation for difficulty. It works beautifully at the scale it was designed for — a handful to dozens of editors through a server, which is exactly our requirement.

OT's trade (Bonus C): battle-proven and efficient, but complex to implement and typically server-coordinated. You buy correctness with implementation difficulty and a coordination point.

Deep Dive 2: CRDTs — Convergence by Construction

The newer approach — Conflict-free Replicated Data Types — attacks the problem from the opposite direction.

The Idea: Make Conflicts Mathematically Impossible

Instead of transforming operations to reconcile them, CRDTs design the data structure so that concurrent operations simply cannot conflict — merging is built into the type itself. Apply operations in any order, on any replica, and they converge by construction, with no transformation step and no central coordinator required.

The trick for text: don't identify a character by its position — give it a unique, stable, orderable identity that never shifts. Position 3 lies when the document changes; a permanent ID between two neighbors never does.

Instead of "insert at position 3" (fragile — positions move),
CRDT says "insert this char with ID between neighbor-X and neighbor-Y"
   → the ID is globally unique and permanently ordered
   → A's "s" and B's "!" get DIFFERENT stable IDs
   → every replica sorts by ID → same order everywhere, any arrival order
   → convergence with NO transform, NO central server needed

What CRDTs Cost

Metadata overhead — every character (or element) carries identity metadata, the structure remembers more than the text itself; naive CRDTs can balloon, modern ones optimize hard, but the tax is real. Tombstones — deleted elements often can't just vanish (a concurrent edit might reference them), they're marked deleted and linger, requiring garbage collection. Conceptually subtler to reason about, though operationally simpler — no transformation matrix to prove correct.

CRDTs' trade (Bonus C): convergence guaranteed by design, peer-to-peer capable, no required central coordinator, bought with metadata overhead and structural complexity. Their superpower is offline and decentralized editing, because they need no server to order operations.

OT vs CRDT, Settled

Given our offline requirement, a CRDT (or a modern hybrid) is the honest choice — offline-for-hours-then-converge is exactly where OT strains and CRDTs shine. If we'd said real-time-only through a server, OT would be equally defensible. This is Bonus C wearing work clothes: the requirement picks the algorithm, not fashion.

Step 4 — Design: The High-Level Architecture

Each editor's device
   │  holds a LOCAL replica — edits apply INSTANTLY (< 16ms, no round-trip)
   │  generates operations (CRDT ops with stable IDs)
   ▼
WebSocket (Part 15's connection layer, reused)
   ▼
COLLABORATION GATEWAYS  (stateful fence — sockets per document, Part 15)
   │        ▲
   │        │  which editors are on which doc → session registry (Redis)
   ▼        │
DOCUMENT SERVICE (per active document)
   │   • receives ops, broadcasts to all other editors (fan-out, Blog 16)
   │   • may assign ordering (OT) or just relay (CRDT)
   │   • maintains the authoritative op log
   ▼
   ├─▶ OPERATION LOG  ← source of truth: the ORDERED STREAM of edits (Bonus H)
   │        │            (the document = the log, replayed)
   │        ▼
   ├─▶ SNAPSHOTS  ── periodic compacted document state (so replay isn't infinite)
   │
   └─▶ events (Part 7) ─▶ presence, notifications (Part 14), version history

Three inherited-and-adapted decisions, named. The source of truth is the operation log, not the document (Bonus H, with a twist) — the document is a derived, rebuildable projection of the ordered op stream, replay the log, get the document; snapshots are a cache of that replay (Part 6); this is the cleanest example in the series of "the events are the truth." The connection layer is Part 15, reused wholesale — stateful gateways holding sockets, thin, everything else stateless; collaboration is chat's connection economy plus a convergence algorithm. Fan-out is Blog 16's shape, miniaturized — every op broadcasts to every editor of that document; a 50-editor doc is a tiny, high-frequency feed.

Deep Dive 3: History, Undo, and Snapshots

The operation log gives history almost for free, but three subtleties bite.

The log grows without bound. A document edited for months accumulates millions of ops. Replaying from zero to open a doc is absurd. The fix is snapshotting: periodically compact the log into a document state, keep recent ops on top. Open = latest snapshot + replay the tail. (Part 8's materialized-view idea; Part 17's precompute, applied to edit history.)

Undo is not "delete the last op." In a shared document, undo must reverse your edit without clobbering the concurrent edits others made after it — undo is itself a new operation (an inverse) that merges through the same convergence machinery. "Undo the last thing" is single-user thinking; collaborative undo is "undo my last thing, consistently, while everyone else's edits stand."

Versioning is snapshots with names. Named versions, restore points, and blame-per-character all fall out of a well-kept op log — you already have the full timeline; versioning is just bookmarking it.

Failure Analysis: Where This Design Bends and Breaks

Two edits truly collide (same position, same instant). The convergence algorithm's entire purpose: OT transforms the coordinates, or CRDT's stable IDs order them, either way, every replica lands on the identical result with both intents preserved. The conflict is resolved, not prevented, and resolution is the normal path, not an error path.

A user edits offline for two hours, then reconnects. Their local replica accumulated a batch of operations; on reconnect, the batch exchanges with the server's stream and both converge. With CRDTs this is clean by construction; the divergence window being long is exactly why the offline requirement pushed us toward them. (Bonus I's replication lag, now measured in hours and convergent rather than merely eventual.)

A gateway holding a document's editors dies. Part 15's answer, reused: sockets drop, clients reconnect (jittered) to another gateway, re-sync from the op log plus latest snapshot. No edits lost — acknowledged ops are in the durable log, and each client's local replica still holds its unsynced ops to replay.

The op log write falls behind under a busy doc. Fan-out for live editing can outrun durable persistence. The design keeps the real-time relay fast (broadcast immediately) while persistence catches up asynchronously, with the guarantee that an op isn't acknowledged as durable until it's logged. A crash before durability replays from clients' local copies.

A malicious client sends malformed or backdated operations. Ops are validated server-side; a client cannot forge another user's identity on an op, and operations reference stable IDs that the server checks (Part 11 — the connection is a pipe, not permission, every op is authorized).

The document gets enormous / the op log explodes. Snapshotting plus tombstone garbage collection (CRDT) keep it bounded. An un-compacted log is this system's version of Part 8's un-lifecycled storage — a slow-growing cost that must be designed against from day one.

Every failure: named, bounded, answered, and the defining one (concurrent conflict) is resolved by the core algorithm, not handled as an exception.

Security and Abuse: The Part Everyone Skips

Collaboration means granting others write access to shared state, the security surface is sharing itself. Design defensively (Part 11): document-level authorization on every operation ("connected to the doc" is not "allowed to edit it," view vs. comment vs. edit permissions are checked per operation, not just at open — Part 15's rule: the socket is a pipe, the op needs authorization). Sharing and access control as first-class design (link sharing, role grants, revocation that takes effect immediately, a removed collaborator's next op is rejected, Bonus I's revocation-lag applies, keep the window tight). Operation authenticity (an op carries its author's verified identity, a client cannot forge edits as another user or rewrite history it didn't make). Presence privacy (cursors and selections reveal what someone is reading, presence respects permissions, you don't appear in a doc you can only view if the product says so). Rate-limiting operations (Part 10) — a runaway client or bot flooding ops per second gets throttled per user/document. History as audit — the op log is also an audit trail: who changed what, when, valuable for trust and required in regulated contexts (Parts 11, 12). No sensitive content in operational logs/telemetry (Part 12) — the document content is the user's private data.

The Observability Plan

Part 12's table, tuned to convergence and real-time feel:

SLO:          99.9% of remote edits visible < 200ms; zero divergence events
Convergence:  DIVERGENCE DETECTIONS (must be ~zero — replicas that disagree
              after sync is a correctness breach; alert on ANY),
              op-transform/merge latency, convergence time after reconnect
Real-time:    local-edit feedback latency, remote-edit propagation delay,
              op fan-out rate per document
Connections:  sockets per gateway, reconnect-success p99 (Part 15), per-doc editor count
Op log:       write latency, log growth rate, snapshot frequency, replay time
Offline:      reconnect batch sizes, offline-convergence success rate
Storage:      tombstone growth, GC effectiveness, snapshot storage
Business:     concurrent editors, docs edited, collaboration session length
Synthetic:    robot editors making concurrent edits + verifying convergence per region

The metric to frame: divergence detections. Like Part 18's oversell count, it should always be zero, two replicas that disagree after synchronizing is not a slow experience, it's a broken guarantee. A synthetic test that makes known-concurrent edits and asserts identical results is the smoke detector for the entire system's correctness.

What Interviewers Actually Probe

"How do you handle two people editing the same spot at once?" — not "last write wins" (that loses data); OT transforms coordinates, or CRDTs use stable IDs, both converge with intent preserved; naming why positions fail (they shift under concurrent edits) is the differentiator. "OT or CRDT — which and why?" — tie it to the requirement: server-mediated real-time favors OT, proven; offline-first/decentralized favors CRDT, converges by construction; the senior answer picks by constraint, not fashion (Bonus C). "Why not just lock the document / use strong consistency?" — because locking means one-writer-at-a-time, which destroys the product; this is the system where strong consistency is deliberately wrong, say that out loud. "What's the source of truth — the document or the edits?" — the ordered operation log; the document is a derived, replayable projection (Bonus H); this reframing signals real understanding. "How does offline editing reconcile?" — local replica accumulates ops, converges on reconnect, CRDTs make the long divergence window clean. "How do you keep the op log from growing forever?" — snapshotting plus tombstone GC. "What breaks first at scale?" — per-document fan-out and op-log growth on hot docs, then connection management across millions of idle docs.

The meta-move: frame correctness as convergence, not single-truth. The candidate who explains that "correct" means all replicas reach the same state via a merge algorithm, rather than one server holding the truth, has understood what makes collaboration a distinct category.

What We Deliberately Didn't Build

A production OT/CRDT implementation — the algorithms are deep specialties with mature libraries, we designed the system around them, not the transformation matrix or the CRDT internals. Rich-text and structural conflict semantics — merging concurrent bold plus link plus table edits is genuinely hard, we designed for text and named where richness multiplies the cases. The spreadsheet/whiteboard variants — same convergence backbone, different data types and conflict rules, a family, not a new problem. Full offline-first sync engines — a robust offline-first product (background sync, partial replicas) is its own build, we required offline convergence and chose the algorithm family that supports it. Peer-to-peer / server-less collaboration — CRDTs enable it, but P2P networking, discovery, and trust are a separate discipline. Access-control UX (sharing flows, granular permissions surfaces) — product surface atop the authorization core we specified.

Common Mistakes Engineers Make with This Problem

"Last write wins" — the reflex that silently deletes someone's work, the anti-pattern the whole field exists to replace. Locking the document for strong consistency — correct data, unusable product, one typist at a time defeats the point. Treating position as stable — positions shift under concurrent edits, coordinates written for one state lie against another, the root of every naive conflict bug. Rolling your own OT transforms casually — every op-pair in every order must be proven convergent, "mostly works" means "silently diverges under load." Making the document the source of truth instead of the op log — you lose replayable history, clean reconnect, and versioning, all of which the log gives free. No snapshotting — the op log that grows forever and takes minutes to replay on open. Ignoring tombstone growth (CRDT) — deleted-but-retained metadata accumulating with no GC, a slow storage leak. Round-tripping to the server per keystroke — the laggy editor that feels broken, local-first application is non-negotiable for feel. Undo as "delete last op" — single-user thinking that clobbers others' concurrent edits, undo is an inverse operation through the merge. Authorizing at connection, not per operation — the viewer who can still write because the socket was a pipe, not a permission. No divergence detection — the correctness breach you can't see, replicas quietly disagreeing with no alarm.

From My Journey: An Architectural Lesson

Collaborative editing is the feature that looks like magic and hides like a monster. Watching several cursors move through a shared document in real time, it seems almost simple, surely everyone's just sending their changes to a server that stitches them together.

Then you try to build it, and the first naive version works flawlessly right up until two people edit the same sentence at the same moment, and one person's words silently vanish, or the two screens quietly stop matching. No error. No crash. Just divergence, and a user asking why their paragraph disappeared.

The realisation that reorganizes the whole problem: there is no single document being edited, there are many copies, diverging constantly, and the system's real job is guaranteeing they converge. Correctness stops meaning "one true state everyone reads" and starts meaning "many states that provably reconcile to the same result, preserving what people meant." That is a genuinely different definition of correct from everything earlier in the series, and once you accept it, the strange machinery (transforming operations, giving characters permanent identities) stops looking exotic and starts looking necessary.

And it's the same trade-offs relocated. Strong consistency versus usability, resolved against strong consistency, uniquely. Real-time feel versus coordination, resolved with local-first edits and background convergence. Simplicity versus offline capability, the requirement that picks OT or CRDT. There was never a design that made concurrent editing simple; only one honest about paying for convergence instead of pretending conflicts don't happen.

The lesson that stuck: a collaborative document is not a file that many people edit. It is many copies that must agree, and the agreement, not the file, is the system.

Key Takeaways

• Collaboration removes the single writer: concurrent edits are the normal case, not a race to prevent.
• "Last write wins" is a data-loss bug — correctness here means convergence, not one true state.
• Positions lie under concurrency: an edit's coordinates are only valid against the state it was written for.
• Operational Transformation reconciles by rewriting operation coordinates against each other — proven, efficient, usually server-coordinated, hard to implement.
• CRDTs make conflicts impossible by construction via stable, orderable element identities — converge with no coordinator, ideal for offline and decentralized editing, at a metadata cost.
• The requirement picks the algorithm: server-mediated real-time favors OT; offline-first favors CRDTs (Bonus C).
• The source of truth is the ordered operation log; the document is a derived, replayable projection (Bonus H) — the series' purest "events are the truth."
• Snapshotting bounds the log; tombstone GC bounds CRDT metadata; undo is an inverse operation, not a deletion.
• Strong consistency is deliberately the wrong answer — locking to get it would destroy the product (Bonus F's dial, turned on purpose).
• Divergence detections must be zero — the correctness metric, like Part 18's oversell count.

Interview Lens

Collaborative editing rewards understanding the shape of conflict resolution over reciting algorithm names.

Lead with why the naive answer fails. "Last write wins loses data, and positions shift under concurrent edits, so an edit's coordinates stop meaning what they meant." Establishing the problem precisely makes OT and CRDTs land as inevitable solutions rather than trivia.

Then frame correctness as convergence. "There's no single truth mid-edit, the goal is that every replica provably reaches the same state, preserving intent." That reframing is the single most senior-sounding move in this problem.

Pick your algorithm by the requirement (offline → CRDT), name the op-log-as-source-of-truth, and volunteer the offline-reconnect and hot-document-fan-out failure walks.

The meta-skill: recognizing that this system redefines "correct." Every prior category protected a single source of truth; this one manufactures agreement among many. The candidate who names that shift understands why collaboration is its own category.

Real-World Engineering Lens

Don't hand-roll OT or CRDTs. These are proven, subtle algorithms with mature libraries, the engineering skill is choosing the right one and building the system around it, not reinventing the transform math.

Model the op log as the source of truth wherever edits accumulate. Even outside editors, audit systems, config changes, any "history of changes," treating the ordered operation stream as truth (with the current state as a derived projection) buys replay, versioning, and clean recovery. This is event sourcing, and collaboration is its sharpest example.

Snapshot anything replayable. An unbounded log is a slow outage; periodic compaction keeps open/replay fast (Part 17's precompute lesson).

Authorize per operation, not per connection, for any real-time write channel, the pipe and the permission are different things (Parts 11, 15).

Test convergence explicitly. A synthetic that makes known-concurrent edits and asserts identical final state is the only way to know your merge is correct, divergence hides silently otherwise.

What's Next

Section 3 has one category left, and it's the layer everything else in this series quietly runs on.

We've built systems that read, deliver, sell, and collaborate. Every one of them assumed a substrate underneath: distributed queues that don't lose messages, storage that survives machine death, analytics that ingest a firehose. Those aren't applications, they're the infrastructure the applications stand on.

Bonus J — Time-Series Databases: When and Why. The analytics and time-series data Part 20 leans on has its own kind of database, one the series has referenced four times without designing. Bonus J answers it before we build the infrastructure that generates it.

After that, Part 20 — Infrastructure Systems: Distributed Queues, Storage, and Analytics. The foundation layer, designed from the inside: how the message brokers, object stores, and data pipelines we've used for nineteen articles are actually built to be reliable.

Question for Readers

Find one place in your system where you resolve conflicting updates with "last write wins."

Whose work silently disappears when that fires, and would anyone even know?

Most systems have at least one. Collaboration just makes the loss visible enough that you have to solve it.