Payments and Transactions

In every other system, a bug is a bad experience.

In a payment system, a bug is a wrong number in someone's bank account — and it doesn't disappear when you fix the code.

It has to be found, explained, and corrected, to the cent, sometimes to a regulator.

Section 3 built application categories. Part 20 revealed that the infrastructure beneath them is "no magic, only primitives combined with discipline." Section 4 now goes deep on the problems where that discipline is tested hardest, and it opens with the one the whole series has been circling.

Money.

Part 7 introduced effectively-once and pointed here. Part 14 built idempotency for OTPs and pointed here. Part 18 previewed the Saga at checkout and pointed here. Every one of those forward references converges now, because payments is where "approximately once" stops being a tolerable engineering compromise and becomes a financial loss, a broken trust, or a legal problem.

This isn't a new set of tools. It's the correctness tools of the entire series — idempotency, consistency, sagas, durability, reconciliation — assembled at their strictest, on the data where being wrong costs the most. Same walkthrough template. The highest stakes in software.

Why Payments Are the Hardest Correctness Problem

Errors are irreversible and visible. A stale feed self-heals; a mis-ranked post is forgotten. A double charge is money that left someone's account — you can refund it, but you cannot make it never have happened, and the customer saw it. Payment errors have permanence and an audience.

The truth spans systems you don't control. Your database holds the order. The card network holds the charge. The bank holds the account. Bonus H's iron law — no transaction spans independent systems — is not an inconvenience here; it's the central problem. You must keep several systems agreeing about money when you can't wrap them in one transaction and can't fully trust any single one's momentary answer.

"Did it happen?" is often unanswerable in the moment. A charge request times out. Did the money move or not? The network doesn't know. Your database doesn't know. Only a designed mechanism — an idempotency key, a reconciliation process — can turn that terrifying uncertainty into a definite answer. Most of payments engineering is manufacturing certainty out of ambiguous failures.

And correctness must be provable, forever. It isn't enough to be right. You must be able to show you were right — every movement of money traceable, auditable, reconstructable, to the cent, months later, for a customer dispute or a regulator. Payments is the only category where "the logs will explain it" is a legal requirement, not a nicety.

A good architect does not ask only, "Did the charge succeed?" A good architect asks, "Can I prove, months from now, exactly what happened to this money — and would I bet my job on the answer?"

Step 1 — Clarify: The Questions Before the Boxes

"Design a payment system" spans a checkout button and a bank core. The separating questions: our scope — integrating a payment processor (Stripe-style), or building the rails (the network/bank layer)? We integrate; building rails is a regulated, multi-year endeavor. Payment types — one-time charges, subscriptions, refunds, payouts to sellers (marketplace)? Money movement — single-direction (customer → us) or multi-party (marketplace: customer → us → seller)? Failure tolerance? (Rhetorical — zero. Money is never "eventually approximately correct.") Compliance scope — do we touch card data, or tokenize it away? (Tokenize; PCI scope is radioactive.) Scale — a single merchant or a platform processing for millions?

Our assumptions, stated (Bonus A's rule):

Scope:        Integrate external processors; WE own idempotency + ledger
Types:        Charges, refunds, payouts (marketplace-capable)
Movement:     Multi-party — customer → platform → seller
Correctness:  Absolute — every cent accounted for, exactly once, provably
Card data:    Tokenized through the processor; never touches our systems
Scale:        Platform-scale, millions of transactions/day

That correctness line is not an NFR among others. It is the requirement, and every design decision below is subordinate to it.

Step 2 — Requirements: The Building Code

Functional: accept a payment; charge through a processor; record every money movement; handle refunds and payouts; reconcile against the processor; expose transaction history; support disputes.

Non-functional, and notice correctness dominates every row (Bonus B, per data type, taken to its limit):

CORRECTNESS (the requirement that owns all others):
  Exactly-once effect      a retry NEVER creates a second charge
  No lost money            an accepted payment is NEVER silently dropped
  No created money         the books always balance — debits == credits, always
  Auditability             every movement traceable and reconstructable, forever

CONSISTENCY:
  Ledger                   strongly consistent — the source of truth for money
  Payment state            a durable state machine surviving every crash (Part 18)

AVAILABILITY:
  Checkout                 99.95%+ — a down payment path is lost revenue
  Degradation             fail CLOSED on ambiguity — never charge when unsure

LATENCY:
  Authorization            seconds (a human is waiting), but correctness > speed

COMPLIANCE & SECURITY:
  Card data                tokenized, never stored (Part 11 — PCI scope minimized)
  Audit trail              immutable, retained per regulation (Parts 8, 11, 12)

Read "fail closed on ambiguity" against the whole prior series. Every other system optimized for availability — serve something, degrade gracefully, partial answer beats a 500. Payments inverts it: when unsure, do nothing and escalate. A missing charge is recoverable; a wrong charge is an incident. This is the deepest philosophical break in the series.

Step 3 — Estimate: The Numbers That Shape the Design

Bonus A's method, and here the surprising truth is that volume is not the hard part:

Assume: platform processing 10M transactions/day

THROUGHPUT (modest, by series standards)
  10M/day ÷ ~100K sec ≈ 100/sec average → ~500/sec peak
  → Part 13 served 4,000 QPS on a URL shortener. Payments is LOWER volume.
    The difficulty was NEVER throughput.

LEDGER WRITES (the real constraint — correctness, not scale)
  Each transaction = multiple ledger entries (double-entry, below)
  → maybe 1,000–2,000 ledger writes/sec at peak — still modest
  → but EVERY ONE must be strongly consistent and balanced. No caching,
    no eventual, no approximation. The constraint is correctness density.

STORAGE (grows forever, by design)
  Ledger + transaction history is APPEND-ONLY and NEVER deleted
  10M/day × several entries × ~1 KB ≈ tens of GB/day, retained for YEARS
  → not a scale problem; a retention-and-audit problem (Part 8)

RECONCILIATION (the hidden workload)
  Every transaction must be matched against the processor's record
  → a continuous batch workload comparing millions of records daily

The verdict flips every prior intuition: payments is a low-volume, maximum-correctness problem. The URL shortener moved more requests per second. What makes payments hard is not load — it's that not one of these modest writes may ever be wrong, and every one must reconcile against an external system's independent record. Correctness density, not throughput, is the whole game.

Step 4 — Design: The High-Level Architecture

The organizing idea: the ledger is the source of truth for money, a durable state machine drives each payment, and everything crossing a boundary is made idempotent and reconciled.

Checkout ──▶ PAYMENT SERVICE (orchestrator)
                │  assigns idempotency key, drives the state machine
                │
                ├─▶ IDEMPOTENCY STORE ── "have I seen this key?" (dedup, Part 14)
                │
                ├─▶ PAYMENT STATE MACHINE ── durable, survives crashes (Part 18)
                │     CREATED→AUTHORIZING→AUTHORIZED→CAPTURED→SETTLED
                │           └────────────────────────→ FAILED / REFUNDED
                │
                ├─▶ EXTERNAL PROCESSOR ── the boundary you don't control
                │     (tokenized card data; idempotency key passed through)
                │
                └─▶ LEDGER ── SOURCE OF TRUTH for money (double-entry, below)
                        │      strongly consistent, append-only, always balanced
                        ▼
                  events (Part 7) ─▶ notifications (Part 14), payouts,
                                     analytics, reconciliation feed
                        ▲
   RECONCILIATION ──────┘  continuously matches ledger vs processor record
   (the process that never sleeps)

Four inherited-and-hardened decisions, named. The ledger is the source of truth (Bonus H, at maximum strictness) — not the order, not the processor, not a cache; the ledger is the one authoritative record of every cent, and everything else is derived from or reconciled against it. The payment state machine (Part 18, load-bearing now) — every transition persisted, so a crash mid-payment is recoverable, never ambiguous; the state machine is how "did it happen?" always has an answer. Idempotency at every boundary (Part 14, everywhere) — the entry point, the processor call, the retry path; a repeated request never doubles an effect. Reconciliation as a permanent citizen (new) — not a cleanup job but a core service, continuously proving the ledger and the external world agree.

Deep Dive 1: Idempotency — Turning "Did It Happen?" Into a Lookup

The scariest moment in payments: you send a charge, and the response never comes back. Timeout. Did the money move?

Retry blindly and you might double-charge. Don't retry and you might have failed to charge a customer who thinks they paid. Both are incidents. Idempotency is the mechanism that makes the retry safe, and it's Part 14's OTP lesson, now guarding money.

The Idempotency Key

Every payment operation carries a key derived from the business intent, not randomly generated, so any retry of the same intent produces the same key:

key = order_8841 + charge_attempt_1   (deterministic, not random)

First request with this key:
   → no record → EXECUTE the charge → store (key → result) → return result

Retry with the SAME key:
   → record exists → return the ORIGINAL result → DO NOT charge again

The timeout is now defanged: retry with the same key, and either you get the original success (it happened) or you execute exactly once (it hadn't). "Did it happen?" became a lookup in the idempotency store.

The Three Subtleties That Separate Real from Naive

The key must be stored durably and checked atomically. Two concurrent retries must not both pass the check — the check-and-claim is a single atomic operation (Part 18's atomic decrement, Part 14's claim-before-act), or you've built a race that double-charges under exactly the concurrency retries create.

The key must cover the exact operation. order_8841 alone is wrong — a legitimate second charge on the same order (a separate purchase) must get a different key. The key binds to the specific intent, or it either blocks valid charges or merges invalid ones.

The processor's own idempotency must be used too. Real processors accept an idempotency key on their API. You pass yours through, so their side also dedupes — belt and suspenders across the boundary, because your store and theirs can each fail independently.

Idempotency is not a feature you add. In payments it is the precondition for retrying anything, and since everything across a boundary must be retryable (Part 7), everything must be idempotent.

Deep Dive 2: The Ledger — Double-Entry as the Source of Truth

This is the idea that separates a payment system from a payments bug. Money is not tracked as balances you update — it's tracked as an immutable log of movements, using a technique accountants have trusted for centuries: double-entry bookkeeping.

The Core Rule: Every Movement Is Two Entries That Sum to Zero

Money never appears or vanishes — it only moves between accounts. So every transaction is recorded as a matched pair (or set) of entries — a debit and a credit — that always balance:

Customer pays ₹1,000 for an order:
  ledger entry 1:  DEBIT   customer_wallet    −1,000
  ledger entry 2:  CREDIT  platform_holding   +1,000
                                              ───────
                          sum of every transaction =  0   ← ALWAYS

Marketplace payout later:
  ledger entry 3:  DEBIT   platform_holding   −1,000
  ledger entry 4:  CREDIT  seller_account     +900
  ledger entry 5:  CREDIT  platform_revenue   +100   (the fee)
                                              ───────
                                       sum  =  0

Two properties fall out, and they are why this technique owns money everywhere. The books must always balance — if the sum of all entries isn't zero, money was created or destroyed, a bug that announces itself immediately; balance is a continuously checkable invariant, not a hope. This is the single most powerful correctness property in the article. Nothing is ever updated, only appended — you never overwrite a balance, you append a new movement; a balance is derived by summing entries (Blog 19's event-sourcing, Part 20's log-as-truth, the ledger is the purest instance in the series). History is complete, immutable, and auditable by construction, because the history is the data.

Why Append-Only Immutable Entries Matter

A mutable balance (balance = balance - 1000) is Part 18's read-then-write race waiting to double-spend, and it destroys history — you can't answer "why is this balance what it is?" An append-only ledger makes every balance explainable (replay the entries), auditable (nothing was hidden or overwritten), and reconstructable (rebuild any account's state at any past moment). The regulator's question — "prove this balance is correct" — is answered by the ledger's structure, not by trust.

The ledger is the source of truth; balances are a derived cache of it (Bonus H, Part 6). That single sentence is the spine of every serious payment system.

Deep Dive 3: Sagas — Correctness Across Services You Can't Transact Across

Part 18 introduced it; here's the full machinery. A payment spans steps across systems that can't share one transaction — reserve, authorize, capture, record, pay out. If a later step fails, you can't roll back a real charge. So instead of rollback, you compensate: every forward step has a designed undo.

Orchestration vs Choreography — the Real Design Choice

The saga's steps must be coordinated, and there are two ways:

ORCHESTRATION (a central coordinator)          CHOREOGRAPHY (events, no coordinator)
──────────────────────────────────            ────────────────────────────────────
Payment orchestrator explicitly calls:         Each service reacts to events:
  → authorize → on success →                     PaymentAuthorized → capture service acts
  → capture   → on success →                     PaymentCaptured   → ledger service acts
  → record    → on failure → compensate          (failure) → compensating events cascade

Pros: clear, debuggable, one place owns          Pros: decoupled, no central bottleneck
      the flow and its compensations             Cons: emergent flow, HARD to trace
Cons: the coordinator is a critical service            and reason about at scale

For payments, orchestration usually wins, because when money is moving, you want one place that knows the exact state of the flow, owns the compensations, and can answer "what happened to this payment?" definitively. Choreography's emergent, hard-to-trace flow is a poor fit for money, however elegant its decoupling. (Bonus C: the requirement — provable correctness and clear ownership — picks orchestration; a high-decoupling, lower-stakes flow might choose choreography.)

Compensation, Not Rollback

FORWARD:  authorize → capture → record in ledger → confirm

FAILURE after capture, before ledger record:
   COMPENSATE: refund the capture (a NEW forward transaction, not an "undo"),
               record BOTH the capture and the refund in the ledger,
               fail the payment cleanly

The crucial point: you cannot un-charge a card, you refund it, and the refund is itself a recorded ledger movement. The ledger shows the charge and the compensating refund; nothing is erased, because money that moved must stay in the history even when it moves back. Compensation preserves the audit trail that rollback would destroy. Every saga step is idempotent (Deep Dive 1), so any step can be safely retried during recovery.

Deep Dive 4: Reconciliation — The Process That Never Sleeps

Here is the part beginners never see and experts never skip. Your ledger says one thing. The processor's records say another. They will disagree, not because of bugs, but because two independent systems processing money across a network always drift: a charge that succeeded on their side but timed out before your ledger recorded it; a refund they processed that your system missed; a settlement that landed a day late.

Reconciliation is the continuous process of comparing your ledger against the external truth and resolving every discrepancy. It is not a cleanup job. It is a core, permanent, always-running service, because in payments, unverified correctness is not correctness.

CONTINUOUSLY:
  fetch processor's transaction records (settlement files, API)
  match each against the ledger by transaction/idempotency key
  for each MISMATCH:
     charge on their side, missing from ledger → record it, investigate
     ledger entry, missing on their side       → hold, investigate, maybe reverse
     amount mismatch                            → flag for human resolution
  surface unresolved discrepancies to a HUMAN queue (money needs judgment)

Three truths reconciliation teaches. Discrepancies are normal, not exceptional — a healthy payment system produces mismatches every day and resolves them; a system with zero mismatches probably isn't checking. The metric isn't "no discrepancies," it's "no unresolved discrepancies past their SLA." Some resolutions need a human — automated matching handles the vast majority, the ambiguous remainder goes to an operations queue, because moving money on a guess is worse than waiting (fail closed, escalate, the checkout philosophy, at the back office). And the idempotency key is the join key — the deterministic key from Deep Dive 1 is what lets you match your record to theirs, idempotency and reconciliation are two ends of the same design.

Reconciliation is how a payment system proves, not assumes, that it never lost or created money. It's the difference between believing the books balance and knowing they do.

Failure Analysis: Where This Design Bends and Breaks

A charge request times out. The idempotency key plus state machine: retry with the same key learns the true outcome, or executes exactly once. The payment sits in AUTHORIZING; a recovery process (or reconciliation) resolves it against the processor. The scariest failure in payments becomes a definite lookup, not a guess.

The service crashes mid-payment. The durable state machine holds the exact last transition; on restart, recovery reads the state and drives the payment forward or compensates. No payment is ever "lost in memory," because nothing important lived only in memory (Part 9, Part 18).

A step fails after money already moved. The saga compensates with a refund, recorded as its own ledger movement, preserving the trail. The books still balance because the charge and its reversal are both entries summing to zero.

The ledger and processor disagree. By design, reconciliation catches it, auto-resolves the clear cases, escalates the ambiguous ones to humans. This isn't a failure the system suffers; it's a failure it's built to continuously absorb.

A double-submit hits checkout. Same idempotency key means the second request returns the first's result, no second charge. The race that concurrency creates is closed by the atomic check-and-claim.

The processor is down. Checkout fails closed — "we couldn't process your payment, try again" — never a silent success or an optimistic charge. Better a retryable failure than an ambiguous money movement. Availability yields to correctness, on purpose.

A balance looks wrong. Because balances are derived from the immutable ledger, the answer is always reconstructable: replay the entries, find the exact movement that's off, trace it to its transaction. Nothing was overwritten, so nothing is unexplainable.

Every failure: named, bounded, and answered by idempotency, the state machine, the ledger, or reconciliation — the four mechanisms that turn money's ambiguity into provable correctness.

Consistency: What Must Be Strong vs Eventual

Even inside a payment system, not everything needs the ledger's strictness, and knowing the boundary is the design (Bonus F, per data type, at the highest stakes):

STRONG / EXACTLY-ONCE (the money core):
  the ledger writes                    — the source of truth, always balanced
  the idempotency check-and-claim      — or concurrency double-charges
  the payment state transitions        — the record that makes recovery possible

EVENTUAL (everything downstream of the money):
  payment-confirmation notifications   — seconds late is fine (Part 14)
  analytics and reporting              — hours-fresh (Bonus H, Bonus J)
  seller payout scheduling             — batched, not instant
  receipt generation, emails           — post-commit events (Part 7)

The rule mirrors Part 18 exactly: strong consistency where the money moves; eventual everywhere downstream, because the downstream work is correct precisely because the money already moved correctly first. The confirmation email being three seconds late harms no one; the ledger entry being wrong harms everyone. Spend strong consistency only where a wrong answer is a financial error, and let the rest be eventual.

What Breaks at Scale

Payments is low-volume, but "low" is relative, and a few things strain even at modest scale.

The idempotency store gets hot. Every payment checks it; it's on the critical path of every charge. It needs to be fast and strongly consistent (the atomic check-and-claim), which is a genuine tension — cache-fast but not eventually-consistent. Size and shard it (Bonus G) as a first-class store, not an afterthought.

A retry storm hits the processor. A processor blip triggers retries across many payments at once (Bonus E's storm, aimed at your revenue partner). Budgeted retries with jittered backoff (Bonus E), and the processor's own idempotency absorbing the duplicates, keep it from amplifying.

A webhook flood comes back. Processors confirm asynchronously via webhooks; a settlement batch can deliver a surge of them. The webhook receiver is its own ingestion path with its own queue (Part 14's receipts-flood lesson, now with money), and every webhook is idempotent, because processors will deliver the same one twice.

The reconciliation backlog grows. If matching falls behind, unverified transactions pile up, a correctness debt, not just a latency one. Reconciliation throughput is a monitored SLA, because "we haven't checked yesterday's money yet" is a real risk.

The ledger write becomes the bottleneck. Strongly-consistent, always-balanced appends can't be sharded as casually as feed data — account balances must stay consistent. Partition by account where possible, but accept that the ledger trades scale for correctness, on purpose.

The theme: payments scales by protecting correctness first and finding scale within that constraint — the inverse of every Section 3 system, which scaled first and bolted on correctness where needed.

Security, Privacy, and Financial Safety

Payments is where a security failure is directly money lost or stolen — the defensive discipline (Part 11) is non-negotiable.

Never store raw card data. Tokenize through the processor; card numbers never touch your database, logs, or memory (Part 11). This is the single most important rule — it shrinks your PCI scope from radioactive to manageable.

Verify webhook signatures. A payment webhook says "this charge succeeded" — an unverified one is an attacker telling you to ship goods for free. Every webhook's signature is cryptographically verified against the processor's key (Part 11) before it's trusted, the long-promised webhook-verification pattern, finally load-bearing.

Secrets in the vault, rotated. Processor API keys, signing keys, webhook secrets — a leaked payment credential is a direct financial breach (Part 11's vault, at its most critical).

Strong authentication and authorization on money operations. Who can issue a refund, trigger a payout, view a ledger — the highest-privilege operations in the system, gated and audited (Part 11).

Immutable audit trail. Every money movement, every state change, every human resolution, logged immutably, retained per regulation (Parts 8, 11, 12). In payments the audit trail isn't observability; it's a legal artifact.

Fail closed, always. The recurring theme is also a security posture: ambiguity resolves to no money moved, never to an optimistic charge.

One boundary, stated plainly: this article is defensive and architectural. It does not cover fraud-detection tactics, card-testing evasion, or anything that could aid misuse — payment safety means building the honest system correctly, not enumerating attacks against it.

Observability for Payments

Part 12's table, and in payments, observability is how you prove correctness, not just watch health:

SLO:           99.95% checkout availability; 100% of transactions reconciled
Correlation:   orderId → paymentIntentId → paymentAttemptId →
               providerTransactionId → ledgerEntryId
               (one payment, traceable end-to-end across every boundary)
Ledger:        LEDGER IMBALANCE ALERT — sum of entries ≠ 0 (must NEVER fire;
               if it does, money was created/destroyed — page immediately)
State machine: payments stuck in AUTHORIZING/PENDING (the money-in-limbo alert)
Idempotency:   duplicate-request rate (retries caught), store latency
Processor:     charge success rate, timeout rate, webhook delivery lag
Reconciliation: UNRESOLVED DISCREPANCY COUNT + age (the correctness SLA),
               auto-match rate, human-queue depth
Business:      payment success rate, refund rate, revenue reconciled vs expected
Synthetic:     a robot payment (in a test mode) end-to-end every minute

Two metrics get the highlight, and both are correctness alarms, not performance ones. Ledger imbalance must never be non-zero; if the books don't balance, money appeared or vanished, and that's the most serious alert a payment system can raise. And unresolved discrepancy count and age is the reconciliation SLA, the measure of how much money is currently unverified, which is the truest health metric a payment system has.

The correlation-ID chain deserves a note: a single payment touches your service, the processor, the ledger, and the webhook path, and tracing it end-to-end (orderId all the way to ledgerEntryId) is what turns a dispute months later into a lookup instead of an investigation. Part 12's discipline, at its highest-stakes application.

What Interviewers Actually Probe

"You charge the card and your server crashes before recording it — what happens?" — the canonical question; idempotency key plus durable state machine plus reconciliation: the charge is recoverable and discoverable, never lost or doubled; this single answer is the whole interview. "How do you prevent double charges?" — idempotency key derived from business intent, atomic check-and-claim, passed through to the processor's idempotency too; naming all three layers is the differentiator. "Same idempotency key, different request body — now what?" — the subtle one: it should be rejected or flagged, not silently processed; the key must bind to the exact operation. "The processor charged but your timeout fired — did it go through?" — retry with the same key to learn the truth, reconciliation as the backstop; "manufacturing certainty from ambiguity" is the concept to name. "How do you track money?" — double-entry ledger, append-only, balances derived, books always sum to zero; candidates who reach for "a balance column you update" have revealed they haven't built this. "What's a saga and why not just a transaction?" — no transaction spans the processor boundary, compensations, not rollback, you refund, you don't un-charge. "How do you know the system is correct?" — reconciliation; the senior answer: "I don't assume it's correct, I continuously prove it against the processor's records." "Webhook says a payment succeeded — do you trust it?" — only after signature verification, an unverified webhook is an attacker.

The meta-move: frame every answer around "exactly-once effect, not exactly-once execution." You cannot guarantee an operation runs exactly once across an unreliable boundary, you guarantee its effect is exactly once, via idempotency, state, and reconciliation. That distinction is the deepest idea in payments, and naming it is what marks someone who has actually built one.

What We Deliberately Didn't Build

The payment rails themselves — card networks, bank settlement, ACH/wire, the regulated core: a multi-year, licensed undertaking; we integrate a processor and own the correctness around it. A full fraud-detection system — real, critical, and a specialty of its own (and out of scope for safety reasons); we place the hooks (a risk pre-check, async deep analysis), not the models. PCI-compliant card handling — we tokenize precisely so we don't build this; touching card data is a compliance burden to avoid, not a feature to add. Multi-currency and FX — genuinely hard (rates, rounding, settlement across currencies), the core ledger discipline holds, but the currency layer is its own design. Crypto/blockchain settlement — a different trust and finality model entirely, out of scope. The accounting/GL product — a ledger feeds accounting, but tax, revenue recognition, and financial reporting are a domain beyond payment processing.

Common Mistakes Engineers Make with Payments

No idempotency key, or a random one — the retry that double-charges; a random key dedupes nothing because the retry generates a new one. Charging before recording locally — the charge that succeeds while your record fails, money taken, no order; record intent first, reconcile always. Trusting the synchronous response only — ignoring that the timeout case is unanswerable without reconciliation, the response you didn't get is the one that hurts. No webhook inbox / no signature verification — trusting an unverified webhook (free goods for attackers), or losing webhooks because there's no durable receiver. One boolean isPaid flag — no state machine, no recovery, a crash mid-payment leaves an unrecoverable unknown; payments needs states, not a checkbox. A mutable balance column — Part 18's race and the destruction of history in one mistake, the double-spend that can't even be explained afterward. No reconciliation — believing the books balance instead of proving it, the unverified correctness that is not correctness. A mutable ledger — editing or deleting entries destroys the audit trail that is the ledger's entire purpose; append-only, always, a correction is a new entry, never an edit. Retry storms into the processor — unbudgeted retries amplifying a processor blip (Bonus E), aimed, this time, at your revenue. Failing open on ambiguity — charging (or shipping) when unsure, the availability instinct from every other system, catastrophically wrong here. Idempotency key that ignores the request body — same key, different amount, silently processed, the subtle bug that charges the wrong number. Treating reconciliation discrepancies as bugs to eliminate — they're normal, the goal is resolution within SLA, not zero.

From My Journey: An Architectural Lesson

Payment integrations look deceptively simple in the demo. You call the processor's API, you get back "success," you mark the order paid. It works the first time, and the second, and the hundredth, and it's easy to conclude that payments is just another API call with a nicer SDK.

Then the failures that don't happen in demos start happening in production. A response that never arrives, leaving you unsure whether money moved. A retry that quietly charges twice. A webhook that arrives before your own record does, or twice, or claiming something your system never initiated. A settlement file that doesn't match your ledger. None of these are exotic; they're Tuesday, at volume.

The realisation that reorganizes everything: the hard part of payments was never charging the money. It was staying correct about money you'd already touched, across systems you don't control, through failures you can't prevent. Every retry, every callback, every state transition, every refund, every reconciliation must point at the same financial truth, and that truth has to survive timeouts, crashes, duplicate deliveries, and disagreements between systems, and still be provable to a regulator months later.

And it turned out to be the whole series, concentrated. Idempotency from the messaging articles. Consistency from the CAP and replication articles. Sagas from commerce. Durability and state machines from infrastructure. The ledger from event sourcing. Reconciliation as the discipline of never trusting a single system's word about money. There was no new magic, only the old tools, held to a standard where being wrong is not a bug report but a financial fact that has to be found and fixed.

The lesson that stuck: a payment system is not correct when the charge succeeds. It is correct when the system can prove what happened after every timeout, retry, and callback.

Key Takeaways

• Payments is a low-volume, maximum-correctness problem — the difficulty was never throughput; it's that not one modest write may ever be wrong, and every one must be provable.
• It inverts the series' instinct: fail closed on ambiguity — a missing charge is recoverable, a wrong charge is an incident.
• Idempotency turns "did it happen?" from a guess into a lookup — a key derived from business intent, checked atomically, passed through to the processor.
• The ledger is the source of truth: double-entry, append-only, balances derived, and the books always sum to zero — a continuously checkable correctness invariant.
• You never update a balance; you append a movement — history is the data, making every balance explainable, auditable, and reconstructable.
• Sagas coordinate across the processor boundary with compensation, not rollback — you refund (a new recorded movement), because you can't un-charge a card. Orchestration wins for money.
• Reconciliation is a permanent, always-running service — discrepancies are normal; the goal is resolution within SLA, because unverified correctness is not correctness.
• Strong consistency only where money moves; everything downstream is eventual — correct because the money already moved correctly.
• Ledger imbalance and unresolved-discrepancy age are the correctness alarms that define payment health; the end-to-end correlation chain makes any dispute a lookup.
• Never store card data, always verify webhook signatures, fail closed — in payments, a security gap is money.

Interview Lens

Payments is the interview where correctness reasoning is the entire evaluation — the structural moves:

Open by naming the real difficulty. "The hard part isn't throughput — it's staying correct across a boundary I don't control, through failures I can't prevent." That single reframing tells the interviewer you understand the problem's actual shape, not just its surface.

Reach for the four mechanisms, always together: idempotency (safe retries), the durable state machine (recoverable crashes), the double-entry ledger (provable money), and reconciliation (verified truth). Every payment question is answered by some combination of those four — a candidate who names them as a system has understood payments.

Volunteer "exactly-once effect, not execution." The deepest idea in the domain: you can't make an operation run once across an unreliable boundary, so you make its effect exactly once. Saying this unprompted is the single most senior signal available.

Then walk the canonical failure — charge succeeds, server crashes — and show it resolving through the four mechanisms into a definite, provable outcome. That walk, more than any definition, is what proves you've built one.

Real-World Engineering Lens

Audit your idempotency keys on every money path. Random or missing keys are double-charges waiting for a retry. Derived from business intent, checked atomically, or it doesn't count.

Model money as a ledger, not a balance column, anywhere your system tracks value (wallets, credits, points, balances). Append-only movements with derived balances buys you correctness, history, and auditability that a mutable column can never provide.

Build reconciliation from day one, not after the first discrepancy. If money crosses a boundary you don't control, a process that continuously proves your records match theirs is not optional, it's how you know you're correct.

Verify every webhook signature, and make every webhook idempotent. External confirmations are attacker-reachable and duplicate-prone; trust them only after cryptographic verification, process them only once.

Fail closed on money. When any part of a payment flow is ambiguous, the safe default is no money moved plus escalation, the opposite of the availability instinct every other system trains into you.

What's Next

We've now built systems that read, deliver, sell, collaborate, carry, and pay, and held each correct under its own hardest conditions. But every one of them made a quiet assumption: that when a dependency failed, something would catch it.

The final piece is that something. Not a category of system, but a discipline that runs through all of them, the deliberate art of designing for the failures you cannot prevent.

Blog 22 — Designing for Failure: Retries, Circuit Breakers, Bulkheads, and Graceful Degradation. Every failure mode this series named in passing — the retry storm, the stampede, the split-brain, the bulkhead, the compensation — gathered into one discipline: building systems that stay standing when the things they depend on don't.

Question for Readers

Find the most important money-or-value operation in your system — a payment, a credit, a balance change.

If you sent it, and the response never came back, could you answer, with certainty, whether it happened? And could you prove it a month later?

If the honest answer is "we'd have to check the logs and guess," you've found the gap payments exists to close. Most systems have it. The best ones turned it into a lookup.