Blog 18
E-Commerce and Marketplaces
Everything before this article could be a little bit wrong.
A stale feed, a double-counted view, a recommendation an hour old — nobody was harmed.
Here, "a little bit wrong" is a customer charged twice, an item sold that doesn't exist, and a support queue that never empties.
Part 17 delivered the heaviest reads on the internet, and got away with approximate counts and eventual freshness the whole time. A view tallied twice cost nothing.
Commerce ends that mercy.
This is the category the entire series has been building toward. Every consistency lesson — Bonus F's ELC trade, Bonus I's replication lag, Part 7's effectively-once, Part 14's idempotency — was rehearsal. E-commerce is where they stop being interesting and start being money. And its defining feature is one most designs get wrong: it is not one system with one consistency model. It is several subsystems, each demanding a different one, inside a single checkout.
Same walkthrough template. The highest correctness stakes in the series.
Why E-Commerce Is a Consistency Battlefield
Different data, opposite requirements, one product. The catalog wants to be fast and cached — stale-tolerant, read-heavy, exactly like a feed. The payment wants to be exact, isolated, and provably once. Putting both under one consistency model guarantees you've over-built one half and under-built the other. This is Bonus F's "per data type" discipline made unavoidable.
Correctness has a body count. Oversell an item and you cancel a real customer's confirmed order. Double-charge and you've stolen money you must refund with interest and apology. Lose an order mid-payment and you've taken money for nothing. These aren't degraded experiences — they're incidents with financial and legal weight.
The money path crosses a boundary you don't control. Payment runs through an external processor with its own latency, its own failures, and its own sense of humor (Part 7's external dependency, now holding your revenue). You cannot make it transactional with your database. That gap is where the hardest design in the article lives.
And it composes the whole series at once. Cached catalog (6), inventory as a hot contended key (Bonus G), orders through queues (7), payment idempotency (14, 21), polyglot stores per subsystem (Bonus H), consistency chosen per boundary (F, I). E-commerce is the integration exam where getting the consistency map right is the whole grade.
A good architect does not ask only, "How do we sell things online?" A good architect asks, "Which parts of this system may lie a little, and which must never lie at all?"
Step 1 — Clarify: The Questions Before the Boxes
"Design an e-commerce platform" spans a single-seller shop and a global marketplace. The separating questions: single seller or marketplace — one inventory owner, or millions of independent sellers? Inventory model — can items oversell (backorder) or is stock strictly limited (one-of-a-kind, tickets)? Flash sales — predictable steady load, or 100,000 people hitting one item at 12:00:00? Payment scope — do we integrate processors, or build payment rails? (We integrate — building rails is Part 21's neighborhood.) Cart persistence — survives across devices and sessions, or ephemeral? Scale — regional store or global marketplace?
Our assumptions, stated (Bonus A's rule):
Model: Marketplace — many sellers, many buyers
Inventory: STRICT — limited stock, overselling is a real failure
Flash sales: YES — synchronized spikes on single hot items
Payments: Integrate external processors (idempotency is OURS to own)
Cart: Persistent, cross-device
Scale: Global, millions of items, high-contention events
That second answer — strict inventory — is the article's hardest thread. A world that can backorder infinitely is easy. Limited stock under contention is where the real design lives.
Step 2 — Requirements: The Building Code
Functional: browse/search a catalog; add to a persistent cart; check inventory; place an order; process payment; confirm; handle failures and refunds.
Non-functional — and notice each subsystem demands its own row (Bonus B, per data type):
CATALOG (read-heavy, stale-tolerant):
Browse latency p99 < 200 ms
Consistency eventual — a price cached for seconds is fine*
Availability 99.9% — browsing must survive
INVENTORY (contended, correctness-critical):
Oversell NEVER — the hard constraint of the whole system
Consistency strong at the decrement — no two buyers get the last unit
ORDER (the durable record):
Consistency strong — an order either exists completely or not at all
Durability an accepted order is NEVER lost
PAYMENT (external, exactly-once):
Idempotency a retry must NEVER double-charge
Consistency order state and payment state must reconcile
CART (per user, forgiving):
Consistency eventual is fine — a cart is a draft, not a commitment
*One asterisk that becomes a whole section: the displayed price can be slightly stale, but the charged price cannot. Hold that — it's the catalog's sharpest trap.
Read the inventory line again. "Oversell: NEVER" is the one requirement that refuses every shortcut the rest of the series taught. No cache, no eventual consistency, no "approximately" survives contact with it.
Step 3 — Estimate: The Numbers That Shape the Design
Bonus A's method, and the interesting number isn't the average, it's the spike:
Assume: 50M DAU, browse-heavy, ~2% convert to a purchase
BROWSE (the bulk — feed-like)
50M × ~20 catalog views/day = 1B views/day
÷ ~100K sec ≈ 10,000/sec avg → ~50,000/sec peak
→ read-dominant, cacheable, CDN-friendly (Parts 6, 17)
ORDERS (rare by comparison, but sacred)
50M × 2% = 1M orders/day
÷ ~100K sec ≈ 10/sec avg
→ tiny volume, maximum correctness. The inverse of browse.
FLASH SALE (where the design actually breaks)
100,000 buyers → ONE item with 1,000 units in stock
→ 100,000 concurrent decrement attempts on a SINGLE inventory row
→ in the first SECOND
↑ THIS is the number the architecture must survive.
Not the throughput — the CONTENTION on one key (Bonus G, weaponized).
The verdict: browse is a feed problem (solved — cache it). Orders are a trickle of sacred writes. And the flash sale is the whole game — not volume, but 100,000 writers fighting over one number that must never go negative. Every prior system feared load; this one fears contention.
Step 4 — Design: The High-Level Architecture
The organizing idea, stated before the diagram: separate subsystems, each with its own store and its own consistency model, stitched together by events. This is Bonus H's polyglot map and Bonus F's per-data consistency, applied as the primary architecture rather than an afterthought.
┌─────────────────────────────────┐
Browse ───▶ CATALOG SERVICE ──▶ cache (Part 6) + search │ EVENTUAL
(read-heavy, CDN-fronted, feed-like) │ (stale-OK)
└─────────────────────────────────┘
┌─────────────────────────────────┐
Add/edit ──▶ CART SERVICE ──▶ per-user store (fast, cheap) │ EVENTUAL
(a draft — forgiving, cross-device) │ (drafts)
└─────────────────────────────────┘
Checkout ──▶ ORDER ORCHESTRATOR ◀── the correctness core
│
├─▶ INVENTORY SERVICE ── strong decrement, no oversell STRONG
│ (the contended hot key — its own section)
│
├─▶ PAYMENT SERVICE ── idempotent, external processor EXACTLY
│ (crosses the boundary you don't control) ONCE
│
└─▶ ORDER STORE ── source of truth, durable, strong STRONG
│
└─▶ events (Part 7) ─▶ notifications (Part 14),
analytics, fulfillment, seller payout… EVENTUAL
Read the right-hand column top to bottom. Five subsystems, four different consistency models, one checkout. That column is the article. The mistake this design exists to prevent is painting the whole thing one color — either making the catalog as expensive as the payment, or (catastrophically) making the payment as loose as the catalog.
Three inherited decisions, named. The consistency map (Bonus F): browse and cart are eventual; inventory, order, and payment are strong or exactly-once. Chosen per subsystem, in writing, before any box. The store map (Bonus H): catalog cache and search are derived; the order store is the source of truth for what was bought; inventory is its own strongly-consistent store. Events stitch, they don't decide (Part 7): once an order is committed, the downstream world (notifications, fulfillment, payouts, analytics) is driven by events and is happily eventual, because the money already changed hands correctly.
Deep Dive 1: Inventory — The Oversell Problem
This is the hardest correctness problem in the series, and it has a deceptively simple statement: 100,000 people, 1,000 units, and not one oversell.
Why the Obvious Approach Fails
The naive flow:
read stock (1) → is it > 0? yes → decrement → write (0)
Under contention, two buyers both read "1 unit left" before either writes "0." Both pass the check. Both decrement. Stock is now -1 — you sold a unit that doesn't exist. This is a race condition, and at 100,000 concurrent attempts it's not a rare edge case; it's a guarantee.
Caching makes it worse, not better: a cached stock count is Bonus I's stale read with money attached. You cannot cache your way out of the oversell problem — the whole point of caching (serve a possibly-stale copy) is exactly what oversell can't tolerate.
The Mechanisms That Actually Work
The fix is atomicity — the read-check-decrement must be one indivisible operation no other transaction can interleave with. Atomic conditional decrement: "decrement only if stock > 0," executed as a single atomic operation the database guarantees is isolated — two buyers can't both succeed, one wins the atomic op, the other gets "sold out"; this is the workhorse. Optimistic concurrency (version/compare-and-set): read stock with a version, on write succeed only if the version hasn't changed, retry on conflict — great when contention is moderate. Pessimistic locking: lock the row, decrement, unlock — correct, but under 100,000 contenders the lock becomes the bottleneck, everyone queues on one row (Bonus G's hot key as a lock hotspot).
Reservation: The Two-Phase Truth
Real checkout isn't instant — the buyer needs time to pay. So inventory is usually reserved, then confirmed:
1. RESERVE at "add to checkout": atomic decrement into a held state,
with a TTL (e.g., 10 minutes)
2. CONFIRM on successful payment: reservation becomes a committed sale
3. RELEASE on payment failure or TTL expiry: the unit returns to stock
The reservation is what prevents the cruelest bug: two people paying for the last unit because inventory was only checked at cart time, not held through checkout. The TTL prevents abandoned carts from locking stock forever (Part 6's TTL discipline, guarding money now).
Taming the Flash-Sale Contention
One atomic row still serializes 100,000 writers through a single point. The scaling answers (Bonus G, applied to a constraint that can't be approximate): partition the stock (split 1,000 units into N buckets of stock across shards, a buyer decrements one bucket, sold-out means all buckets are empty — spreads the contention at the cost of "sold out" being slightly harder to declare); queue the contenders (put purchase attempts through a queue, Part 7, and process against inventory at a sustainable rate — 100,000 requests become an orderly line, not a stampede; buyers get "you're in the queue," which is honest and survivable); and admission control (only admit as many checkout attempts as there's plausibly stock for, reject the rest early with "sold out" before they contend — the fastest write is the one you never make).
The through-line: inventory is the one subsystem where every trick from the rest of the series — cache it, make it eventual, approximate it — is forbidden. Correctness first; then scale the correctness.
Deep Dive 2: Orders and Payments — Exactly-Once Across a Boundary
Now the subsystem where "approximately once" is a lawsuit.
The Core Problem: Two Systems, No Shared Transaction
The order lives in your database. The charge lives in the processor's. Bonus H's iron law applies: no transaction spans two independent systems. You cannot atomically "create the order AND charge the card." Something can always fail between them, and every failure mode is a real incident:
Charge succeeds, order write fails → customer paid, no order (worst)
Order writes, charge never sent → order with no money
Charge sent, response lost to timeout → did it go through? retry = double charge?
Idempotency: The Answer to "Did It Go Through?"
Part 14 taught this; here it's load-bearing. Every payment attempt carries an idempotency key, derived from the order, not random, so a retry of the same attempt is recognized and collapsed:
Client/orchestrator → processor: "charge order_8841, key=pay_8841_v1"
processor already saw pay_8841_v1? → return the ORIGINAL result, no new charge
first time? → charge once, remember the key
The timeout case, the scariest one, is now safe: retry with the same key, and either you learn the original succeeded or it charges exactly once. The idempotency key turns "did it go through?" from a guess into a lookup. (Part 21 builds the full payment machinery; this is the system-level shape.)
The Saga: A Transaction That Spans Services
Checkout is a sequence of commitments across services that can't share one transaction: reserve inventory → charge payment → confirm order. If a later step fails, the earlier ones must be undone, but you can't roll back a real credit-card charge. So instead of rollback, you compensate:
FORWARD: reserve stock → charge card → confirm order → emit events
FAILURE at "charge card":
COMPENSATE: release the stock reservation, fail the order cleanly
FAILURE after charge, at "confirm order":
COMPENSATE: refund the charge, release stock, notify the customer
This is the Saga pattern: a long-running business transaction made of local steps, each with a compensating action, coordinated by the order orchestrator. It trades ACID's automatic rollback for explicit, designed undo, because across services, undo is the only rollback you get. (Part 21 goes deep; the system point is that the orchestrator owns the sequence and its compensations, and every step is idempotent so it can be safely retried.)
The Order Is the Source of Truth, with a State Machine
The order record ties it together — a durable state machine that survives every crash:
CREATED → INVENTORY_RESERVED → PAYMENT_PENDING → PAID → CONFIRMED
│ │ │
└──────────────┴────────────────────┴──▶ FAILED / CANCELLED (with compensation)
Because the state is persisted at every transition, a crash mid-checkout is recoverable: a reconciliation process finds orders stuck in PAYMENT_PENDING, asks the processor "what actually happened to key pay_8841_v1?", and drives them forward or compensates. The order state machine is how the system stays correct across failures it can't prevent, Bonus I's "recover" answer, made concrete.
Failure Analysis: Where This Design Bends and Breaks
Two buyers race for the last unit. The atomic conditional decrement lets exactly one win; the other gets an honest "sold out." The race the naive design guarantees is the race this design makes impossible. Correctness by construction.
A flash sale sends 100,000 attempts in one second. Queue plus admission control plus partitioned stock convert the stampede into an orderly, survivable line. The metric that matters is fairness and correctness, not raw throughput — nobody oversold, everyone got a real answer.
Payment succeeds but the confirmation response times out. Idempotency key plus order state machine: the reconciliation process queries the processor with the same key, learns the charge succeeded, and drives the order to PAID. No double charge, no lost order. The scariest failure becomes a lookup.
Payment fails after inventory was reserved. The Saga compensates: release the reservation (or let its TTL expire), fail the order cleanly, notify the buyer. Stock returns to the pool; no unit is stranded.
The catalog cache serves a stale price. Fine for display, but the charge uses the price captured at order creation, re-validated at checkout, never the cached browse price. The stale-price trap is disarmed by never trusting the display value for money.
The order store is unreachable at checkout. Checkout fails closed — better to tell the buyer "try again" than to take money without a durable order. The order write is the commit point; nothing irreversible happens before it's safe.
Downstream (fulfillment, payout, email) fails. These are post-commit events (Part 7), the money already moved correctly. They queue, retry, hit a DLQ (Part 14), and catch up. Eventual consistency is correct here precisely because it's downstream of the money.
Every failure: named, bounded, answered, and the ones that touch money are answered by idempotency and a persisted state machine, never by hope.
Security and Abuse: The Part Everyone Skips
Commerce is where security failures convert directly into money lost. Design defensively (Part 11): never trust client-supplied prices or totals (the price and total are computed and re-validated server-side at checkout from the catalog source of truth — a request that says "total: $0.01" is testimony, not evidence, Part 11's rule, with a cart attached). Authorize every step against the actor (a user can only check out their cart, view their orders, use their saved payment methods — object-level authorization, not just "logged in," the classic e-commerce IDOR risk). Payment data stays out of your system (tokenize through the processor, card numbers never touch your database or logs — Part 11's secrets discipline, PCI scope minimization is the reason). Idempotency keys are also abuse defense — they blunt double-submit and replay of purchase requests. Rate-limit checkout and inventory endpoints (Part 10) — flash-sale bots and inventory-scraping scripts get throttled per account, admission control doubles as abuse control. Fraud checks belong on the async path where possible — a fast risk pre-check inline, deeper analysis post-authorization, so fraud scoring never sits on the critical latency budget. Seller-side trust (marketplace) — sellers are semi-trusted actors: listing validation, payout controls, and audit trails on catalog and price changes (Parts 11, 12).
The Observability Plan
Part 12's table, tuned to a system where the business metrics are the health metrics:
SLO: 99.9% of checkouts complete correctly; browse p99 < 200 ms
Catalog: browse latency, cache hit rate, search latency (feed-like)
Inventory: decrement contention, reservation TTL expiries,
OVERSELL COUNT (must be zero — alert on ANY),
stock-partition balance (Bonus G)
Orders: state-machine distribution, orders stuck in PAYMENT_PENDING (alert),
checkout success rate, order write latency
Payments: charge success rate, idempotency-key hits (duplicate attempts caught),
reconciliation actions, refund/compensation rate
Saga: compensation frequency, orphaned reservations, stuck sagas
Business: conversion rate, cart abandonment, revenue/min, flash-sale fairness
Synthetic: robot buyer completing a full checkout every minute per region
Two metrics get the highlight. Oversell count is the only metric in the series that should always be zero — any non-zero value is a correctness breach, not a degradation, and pages immediately. And orders stuck in PAYMENT_PENDING is the early-warning line for the money boundary: a rising count means reconciliation is falling behind, and every stuck order is a customer in limbo.
What Interviewers Actually Probe
"How do you prevent overselling?" — atomic conditional decrement, reserve-then-confirm with a TTL, and never a cached stock count; say "you can't cache your way out of oversell" and you've shown you understand the constraint, not just a technique. "Payment succeeds but your server crashes before recording it — now what?" — idempotency key plus persisted order state machine plus reconciliation querying the processor with the same key; this single answer separates senior from mid. "How do you handle a flash sale on one item?" — queue plus admission control plus partitioned stock, contention, not throughput, is the enemy, name that distinction explicitly. "Different parts need different consistency — walk me through it." — the consistency map: catalog/cart eventual, inventory/order strong, payment exactly-once; leading with "it's not one consistency model" is the answer that lands. "How do you undo a checkout that failed halfway?" — Saga with compensations, you can't roll back a charge, so you refund; explicit undo, not ACID rollback. "Can you cache the price?" — display yes, charge no; the stale-price trap. "What breaks first at 10×?" — inventory contention on hot items, then the payment reconciliation backlog, each with its metric.
The meta-move: refuse to give the system one consistency model. Every strong candidate answer starts by decomposing — this part eventual, that part strong, this part exactly-once. Candidates who paint it one color have missed what makes commerce hard.
What We Deliberately Didn't Build
The payment rails themselves — building a processor (card networks, settlement, ledgers) is Part 21's neighborhood, we integrate one and own the idempotency around it. Full search and recommendations — catalog search is Blog 16/17 machinery reused, the ranking model is its own discipline. Tax, shipping, and fulfillment logistics — real, complex, largely downstream of the money, post-commit event consumers, not checkout-critical. The full fraud system — we placed the hooks (inline pre-check, async deep analysis), the models are a specialty. Marketplace seller tooling — onboarding, payouts, dispute resolution: a whole product surface upstream of the buying flow. Multi-currency and international settlement — genuinely hard, deferred, the core consistency design doesn't change shape.
Common Mistakes Engineers Make with This Problem
One consistency model for the whole system — the root mistake, over-builds the catalog, under-builds the payment, or both. Caching inventory counts — a stale stock number is an oversell with a delay, the one thing caching must never touch. Read-then-write inventory without atomicity — the race condition that guarantees oversell under contention. Checking stock at cart time, not holding it through checkout — two people paying for the last unit, no reservation, no protection. Random or missing payment idempotency keys — the timeout that becomes a double charge. Assuming order-plus-charge is one transaction — it spans two systems, it never was atomic, design the gap or discover it in production. No order state machine — a crash mid-checkout with no way to recover the truth, money in limbo, unrecoverable. Rolling back a charge instead of compensating — you can't un-charge a card, refund is the only undo across the boundary. Trusting client-supplied prices or totals — the $0.01 order, server-side re-validation is non-negotiable. Fraud checks on the synchronous critical path — blocking checkout latency on deep analysis that belongs async. No oversell/stuck-order alerting — the two metrics that must never drift, unwatched. Making downstream (email, fulfillment) synchronous — blocking the commit on work that should be post-commit events.
From My Journey: An Architectural Lesson
E-commerce looks, from the outside, like the most ordinary system of all: a catalog, a cart, a checkout button. Many engineers design it as one application over one database, and for a small shop with a handful of orders a day, that works — the race conditions never fire, the payment never times out, the flash sale never comes.
Then scale and money arrive together, and the ordinary system reveals its teeth.
The realisation that reorganizes everything: e-commerce is not one system, it is a federation of subsystems that disagree about what "correct" means, and the design's real job is honoring each one's definition without letting them contaminate each other. The catalog wants to be a fast, stale-tolerant feed. The inventory wants to be a strict, contended, uncacheable counter. The payment wants to be an exactly-once operation across a boundary you don't own. Force them into one model and you fail half of them.
And every hard part is a consistency decision with money attached. Overselling is a failure of atomicity. Double-charging is a failure of idempotency. A lost order is a failure of durability across a boundary. A stale price charged is a failure of trusting the wrong copy. None of them is a throughput problem — they're all correctness problems, and correctness under contention and across boundaries is the whole discipline.
The lesson that stuck: in commerce, the system is not judged by how fast it sells. It is judged by whether every rupee, every unit, and every order is accounted for exactly once.
Key Takeaways
- E-commerce is not one system with one consistency model — it's several subsystems, each demanding its own, inside a single checkout.
- The consistency map is the architecture: catalog and cart eventual; inventory and order strong; payment exactly-once — chosen per subsystem, in writing (Bonus F).
- Overselling is a race condition; you cannot cache your way out of it — atomic conditional decrement plus reserve-then-confirm with a TTL is the fix.
- Flash sales are a contention problem, not a throughput problem: queue, admission control, and partitioned stock tame 100,000 writers on one key (Bonus G).
- No transaction spans your database and the payment processor — idempotency keys turn "did it go through?" from a guess into a lookup (Part 14).
- Checkout is a Saga: forward steps with compensating undos, because you can't roll back a real charge — you refund it.
- The order is a persisted state machine — the source of truth that stays correct across crashes via reconciliation (Bonus I's "recover").
- Display prices can be stale; charged prices never — re-validate server-side, never trust client totals.
- Post-commit work (fulfillment, email, payout) is eventual and event-driven — correct because it's downstream of the money (Part 7).
- Oversell count and stuck-order count are the metrics that must never drift — alert on any non-zero.
Interview Lens
E-commerce is the interview where decomposition itself is the skill being tested.
Open by refusing one consistency model. "This isn't one system — the catalog is a stale-tolerant feed, inventory is a strongly-consistent contended counter, payment is exactly-once across an external boundary. Let me design each to its own requirement." That opening sentence signals more seniority than any single mechanism, because it reframes the whole problem correctly.
Then go deep where the stakes are: oversell (atomicity, never caching), the payment boundary (idempotency plus state machine plus Saga), and the flash sale (contention, not throughput). Volunteer the crash-after-charge failure walk unprompted — it's the scenario that most reveals whether you've actually built this.
The meta-skill: matching each subsystem to its consistency model and defending the boundaries between them. The candidate who says "I'd make the catalog eventual and the payment exactly-once, and here's how I keep them from contaminating each other" has understood commerce structurally.
Real-World Engineering Lens
Draw the consistency map before the architecture. For any system touching money or limited resources: list each subsystem and its consistency requirement first. The map prevents both over-building and the catastrophic under-build.
Hunt for read-then-write races on anything scarce. Inventory, seats, coupon uses, rate limits, credits — anywhere a count must not go negative, the atomic-decrement lesson applies. These races hide in more systems than commerce.
Make every money operation idempotent and every idempotency key derived from the business fact. Random keys dedupe nothing; the retry that double-charges is the bug idempotency exists to kill.
Persist state machines for multi-step money flows. If a process crosses a boundary you don't control, a durable state machine plus reconciliation is how you stay correct through failures you can't prevent.
Alert on the metrics that must be zero. Oversell count, stuck-in-pending orders, double-charge count — these aren't degradations to trend; they're breaches to page on.
What's Next
We've held money correct across services and boundaries. But we did it with a single writer as the source of truth per record — one inventory count, one order, one owner.
The next category removes even that: many people editing the same document at the same time, every keystroke a change, no single writer — where "correct" means everyone's edits survive and converge, and the conflict is the normal case, not the failure.
Blog 19 — Collaboration Tools: Real-time Sync, Versioning, and Conflict Resolution. Where concurrent writers stop taking turns — operational transforms, CRDTs, and the hardest consistency question yet: what does "the document" even mean when ten people are typing into it at once?
Question for Readers
Find one place in your system where a count must never go wrong — inventory, credits, seats, coupon uses, rate limits.
Is the check-and-update atomic, or is it a read-then-write with a race hiding inside?
Under enough concurrency, that race stops being rare and becomes a guarantee. Commerce just makes the bill arrive faster.