Observability

Every system fails. The only real question is whether it fails articulately.

Look back at this series and notice a pattern. Nearly every article ended with the same kind of instruction: watch the cache hit rate (Part 6). Watch consumer lag and the DLQ (Part 7). Monitor replication lag (Bonus I). Put egress on the dashboard (Part 8). Per-shard p99s, not averages (Bonus G). Per-instance latency, or zombies hide (Bonus E). Instrument the front door (Part 10). You cannot investigate what you never recorded (Part 11).

Twelve articles of homework, all pointing at the same missing discipline. Today it gets its name, and its design principles.

Because observability is not a dashboard bolted on after deployment. It is the architectural requirement that, when this system fails at 2 AM, the evidence to understand the failure already exists — emitted by design, not reconstructed by guesswork.

The Simple Mental Model: The System's Nervous System

Observability is a nervous system. Your body doesn't wait for a doctor to open it up to know something is wrong. Pain, pulse, temperature, breathing, reflexes — the body reports on itself, continuously, and a doctor reads those reports to reconstruct what's happening inside.

Map it across: metrics are vital signs — pulse, temperature, numbers over time. Logs are the detailed notes — what happened, when, with context. Traces are the path the request took — the journey through the body. Alerts are pain signals — interrupts that demand attention. Dashboards are the medical charts — the readable summary at the bedside. Runbooks are treatment plans — what to do when a known pain appears. Audit logs are legal memory — the record that survives for the inquiry.

And the sentence the whole article stands on: a system without observability may still be running, but it cannot explain itself. And a system that can't explain itself turns every incident into an argument.

Monitoring vs Observability

Two words used interchangeably. They're not the same job.

Monitoring answers: "Is the known thing healthy?" — predefined checks on predefined signals. CPU below 80%? Disk not full? Process alive?

Observability answers: "Can we understand an unknown failure using the evidence the system emits?" — the ability to investigate problems you never predicted.

Watch the difference in action. Users are complaining. Monitoring reports:

CPU: fine.  Memory: fine.  Disk: fine.  All checks green.

Observability reconstructs the story: p99 API latency has tripled (Part 10's front-door metrics), requests are queuing on database connections (Part 9's pool math, live), one shard is answering slowly (Bonus G's hotspot), the cache hit rate quietly dropped this morning (Part 6), queue retries are climbing (Part 7), and one client is sending ten times its normal traffic (Part 10's per-client usage).

Notice: you've already met every one of those signals. They were never optional decorations. They were this article, arriving in installments.

Monitoring is necessary and not sufficient. Dashboards show symptoms; observability reconstructs stories.

The Three Pillars: Metrics, Logs, and Traces

The evidence a system emits comes in three shapes, plus two relatives worth naming.

Metrics: Numbers Over Time

Request count. Latency. Error rate. CPU. Queue depth. Cache hit rate. Cheap to store, fast to query, perfect for trends, dashboards, and alerts. Metrics tell you that something changed and how much — rarely why.

(A quiet aside: metrics are time-series data, and storing millions of measurements per second is its own database problem, with purpose-built engines. That story belongs to a future bonus article on time-series databases.)

Logs: Events with Context

The detailed notes: this specific thing happened, here, with these details. Logs carry the why — but only if they're structured (a whole section below) and only if they carry the request's identity so they can be joined into a story.

Traces: The Request's Journey

In a distributed system, one user request touches many services. A trace records that journey — every hop, every span, every millisecond spent:

TRACE req_7f3a9c ................. total 1,840 ms
├─ gateway ........................ 12 ms
├─ auth-service ................... 38 ms
├─ order-api ................... 1,790 ms   ← the suspect
│   ├─ cache lookup ................ 2 ms  (miss)
│   ├─ db query ................ 1,730 ms   ← found it
│   └─ publish event ............... 9 ms
└─ response ........................ 8 ms

Thirty seconds of reading replaces an afternoon of cross-team Slack archaeology. This is distributed tracing — Part 7's distributed-timeline problem, finally given its instrument.

The Relatives: Events and Audit Logs

Events are discrete facts that something happened: a deploy, a config change, an autoscaling action, a failover. Overlay them on your metric charts and half your mysteries solve themselves ("latency jumped… at the exact minute of the deploy").

Audit logs are Part 11's legal memory: who did what, when, to which sensitive thing. Not for debugging — for accountability, compliance, and the investigation that arrives months later.

The pillars are strongest together: the metric says something's wrong, the trace says where, the log says why.

The Signals Every System Should Emit

Three standard checklists, each fitted to a different question:

The pattern to internalize: RED looks at the work arriving (requests), USE looks at the thing doing the work (CPU, memory, pools, disks), and the golden signals wrap both around the user's experience.

Start every new service with RED. Start every resource with USE. If you emit nothing else, emit these.

Why Averages Lie: p95 and p99

The most dangerous chart in engineering is a healthy-looking average. Run the numbers:

95 users → 100 ms
 5 users → 5,000 ms

Average = (95×100 + 5×5,000) / 100 = 345 ms   ← "looks okay"
p95     = ~100 ms
p99     = 5,000 ms                             ← five seconds of pain

The average politely blends five suffering users into a number nobody alerts on. Percentiles refuse to blend: p95 says "95% of users get at most this," p99 exposes the tail, where the real problems live.

And this series has already shown you who lives in the tail: Bonus G's hotspot shard, quietly slow while its siblings idle. Bonus E's zombie server, ruining every Nth request while passing health checks. Part 9's saturated connection pool, punishing the unlucky. Part 10's one expensive query pattern.

Averages are where those problems hide. Percentiles are where they're found. Chart p95/p99 by default; treat the average as trivia.

Correlation IDs: Turning Many Logs into One Story

One user taps "Place Order." The request touches gateway, auth, order-api, cache, database, broker, worker, storage. Eight components. Eight log streams. One user's problem.

Without a correlation ID, that's eight disconnected diaries — and debugging means matching timestamps across services at 2 AM, by hand, wrongly.

With one, the story assembles itself: the gateway stamps a single ID on the request (Part 10's job), every service and, critically, every message carries it onward (Part 7's discipline, because the async hops are where stories usually break), and one search reconstructs the entire distributed timeline:

search: requestId = req_7f3a9c

→ 14:02:11 gateway    accepted
→ 14:02:11 auth       token valid
→ 14:02:13 order-api  db query slow (1,730 ms)
→ 14:02:13 broker     OrderPlaced published
→ 14:07:40 worker     invoice FAILED → retry → DLQ

One ID. One story. Including the failure that happened five minutes after the user left happy — the async failure Part 7 warned nobody would be on the page to see.

The rule: stamp the ID at the front door; drop it nowhere — not across service calls, not into queues, not into background jobs.

Structured Logging: Logs That Machines Can Understand

The difference between a log and evidence. Bad log:

Something went wrong

Good log:

{
  "level": "error",
  "service": "order-api",
  "requestId": "req_123",
  "userId": "user_5521",
  "tenantId": "tenant_a",
  "operation": "create_order",
  "errorCode": "PAYMENT_TIMEOUT",
  "latencyMs": 1840
}

The first is a sigh. The second is queryable: filter by tenant, aggregate by errorCode, join by requestId. Structured logs turn a text pile into a dataset, which is the entire point, because at scale nobody reads logs — they query them.

The disciplines that keep them useful: stable field names (errorCode today, errorCode forever — renamed fields orphan every saved query), log levels used honestly (debug for development detail, info for normal milestones, warn for degraded but coping, error for broken, act — a system that logs everything at error has no alarm left to ring), no secrets, ever (tokens, passwords, keys — Part 11's rule; a credential in a log is a credential with a distribution list, and personal data gets the same caution), and sampling for the flood (keep 100% of errors and slow requests, sample the millions of identical happy-path entries — all the evidence, a fraction of the bill).

A Word on Cardinality

One subtlety that separates cheap telemetry from a horror-story invoice: high-cardinality dimensions — fields with millions of possible values, like userId or requestId.

In logs and traces, they're gold: that's exactly what you search by. As metric labels, they're a bomb: a metric labeled per-user explodes into millions of separate time series, and metric systems bill by series. The rule of thumb: metrics get low-cardinality labels (service, endpoint, status class); the who-exactly questions belong to logs and traces.

SLIs, SLOs, and SLAs in Plain English

Three acronyms that turn "reliability" from a vibe into a number.

SLI — what you measure. The indicator: request success rate, p99 latency, availability. Pick SLIs that track user experience, not machine comfort.

SLO — your internal target. "99.9% of checkout requests succeed within 500 ms." A number your team commits to, measured by the SLI.

SLA — the external promise. The contractual version, with refunds and lawyers attached. Your SLO should always be stricter than your SLA — the gap is your safety margin.

Why SLOs earn their keep: they settle the oldest argument in engineering — features vs. reliability — with arithmetic instead of volume. Within target? Ship features. Burning through your allowance of failure? Reliability work outranks the roadmap this sprint, and nobody has to win a meeting to prove it.

An SLO is a pre-negotiated peace treaty between "move fast" and "don't break things."

Alerts, Alert Fatigue, and Runbooks

An alert is a claim on a human's attention at 3 AM. Price it accordingly.

Good alerts are actionable (a human can do something), owned (a named team answers), tied to user impact (SLO burning, checkout failing — not "CPU 81%"), severity-graded (page vs. ticket), and runbook-linked.

Bad alerts are noisy, unactionable, infrastructure-only with no user consequence, owned by nobody, or firing on symptoms no one understands. Every one of them costs the same thing: trust.

Because alert fatigue is the real killer: after fifty false pages, the fifty-first — the real one — gets snoozed. A noisy alert doesn't just waste time; it trains humans to ignore the alarm. Deleting a useless alert is a reliability improvement.

And every page deserves a runbook, the treatment plan from the mental model:

WHAT happened        — what this alert means in user terms
VERIFY               — how to confirm it's real (which query, which chart)
DASHBOARD            — the one link to open first
COMMON CAUSES        — the usual suspects, in order of likelihood
SAFE MITIGATIONS     — steps that help and can't make it worse
ESCALATION           — who to wake next, and when

That's also the honest definition of on-call readiness: not heroism — alerts worth waking for, runbooks that shortcut the panic, and drills that tested both before the real night.

Observability Across the System

Now the convergence. Every layer this series built, and what it must report — the homework from eleven articles, on one chart:

Print it. This table is the observability review checklist for every design you'll do from Part 13 onward.

Business Metrics Are Also Observability

Here's the failure mode all that infrastructure telemetry can't catch: every technical dashboard is green, and no one has completed a checkout in twenty minutes.

Maybe a payment provider is quietly declining everything. Maybe a frontend deploy broke the button. The servers are healthy. The business is down.

So the journeys that matter get their own vital signs: checkout success rate, payment failure rate, order placement rate, login success rate, search zero-result rate, notification delivery rate, video processing completion rate, support-ticket volume, conversion.

These are observability, not "analytics for later," because they alert on the only definition of health that counts: the system is healthy only if users can complete the journeys that matter. Everything else is machinery.

And Sometimes: Be Your Own User

Internal telemetry is white-box monitoring — reading the system's own instruments. Its blind spot: everything outside the system. Expired certificates, DNS trouble, a broken third-party script — invisible from inside, obvious from a browser.

Black-box monitoring probes from outside, like a user would. Its sharpest form is synthetic monitoring: a robot customer running the critical journey — load page, log in, add to cart, check out — every minute, from several regions, forever.

When the robot fails, you learn about the 2 AM breakage at 2:01, instead of from the morning's angriest ticket.

Cost Observability

One more axis, because a system can be technically healthy and financially hemorrhaging, and no pager fires for a bill.

The usual suspects, all previously met: storage growth with no lifecycle rules, orphaned objects accruing rent (Part 8); egress, Part 8's classic bill shock, still undefeated; over-scaling, Part 9's outage-shaped problem that only finance graphs; expensive queries running frequently, invisibly; excessive logs, telemetry so verbose it becomes its own cost center; high-cardinality metric labels, the cardinality bomb, itemized; unused resources, the staging fleet from a project that shipped last year; retry storms, Bonus E's amplifier, now with per-request pricing; and cold-storage retrieval surprises, the archive that charged admission.

The fix is the same discipline as everything above: put cost on a dashboard, trend it, alert on anomalies, and give each line an owner. A bill is just a metric with a currency symbol.

Common Mistakes Engineers Make with Observability

Logs but no metrics — detail with no trends, you can autopsy, never anticipate. Metrics but no correlation IDs — you know something's wrong, the story won't assemble. Dashboards with no owners — wall art. Alerts with no runbooks — a 3 AM riddle. Alerting on CPU, not user impact — the machine's comfort, prioritized over the customer's. Ignoring p95/p99, relying on averages — the tail is where the users are suffering. Logging sensitive data — Part 11's rule, broken at scale. No audit trail — the investigation with no evidence. No business metrics — green dashboards, silent checkout. High-cardinality labels everywhere — the invoice arrives before the insight. No sampling strategy — paying full price for a billion identical happy-path lines. No cost visibility — financially unhealthy, technically proud. No observability in async workers — Part 7's delayed failures, happening to no witnesses. No DLQ alerts — customer bugs, aging quietly in a waiting room. No replication-lag monitoring — Bonus I's window, unmeasured and therefore unmanaged. Never testing observability — the first time you check whether the evidence exists should be a drill, not the incident.

Applying C.R.E.D to Observability

Clarify — What are the failure modes this system can have, and what would we need to see to diagnose each? Which user journeys define "healthy"?

Requirements — SLIs and SLOs per journey; retention and audit obligations; what must never appear in logs.

Estimate — Telemetry volume and its cost; cardinality budgets; alert load a human on-call can actually absorb.

Design — The RED/USE baseline per component, correlation-ID propagation, the dashboards and their owners, the alerts and their runbooks — designed with the system, in the same review.

A good architect does not ask only, "What logs should we add?" A good architect asks, "When this system fails at 2 AM, what evidence will help us understand the failure without guessing?"

From My Journey: An Architectural Lesson

In many systems, observability starts late, and always in the same order. Logs get added when something breaks. Dashboards get created after an outage. Alerts appear after users complain. Each incident donates one more instrument to the panel, after the fact.

The pattern has a cruel structure: by the time the incident happens, the evidence needed to understand it was never emitted. The team knows something is wrong — the complaints make that clear — but can't quickly answer the four questions that matter: where is it wrong, why, who is affected, and is it getting worse? So the incident channel fills with the only thing available: theories.

The realisation that reframes the discipline: observability was never about collecting everything. Collect everything and you drown in noise and cost. It's about collecting the right evidence, chosen in advance, by asking what the system would need to say to explain itself under stress.

And the trade is real in both directions. More telemetry buys visibility and costs money, noise, and privacy risk. Less telemetry saves the bill and forces the team to debug by guessing, which is also a cost, paid at 2 AM, with interest.

The lesson that stuck: observability must be designed with the system, not attached after failure.

If the system cannot explain its failure, the team will explain it with guesses.

Key Takeaways

• Observability is a design requirement: the evidence for the 2 AM incident must be emitted before the incident.
• Monitoring checks known signals; observability lets you investigate unknown failures. You need both.
• Three pillars, one story: metrics say something's wrong, traces say where, logs say why — joined by a correlation ID stamped at the front door and dropped nowhere.
• RED for services, USE for resources, golden signals for the user — the default instrumentation of every component.
• Averages blend the suffering away; p95/p99 is where hotspots, zombies, and saturated pools are found.
• Structured logs are a queryable dataset: stable fields, honest levels, sampled happy paths, zero secrets — and high-cardinality belongs in logs and traces, never in metric labels.
• SLOs turn the features-vs-reliability argument into arithmetic; SLAs are the stricter promise's contractual shadow.
• Every alert is a claim on human sleep: actionable, owned, user-impact-tied, runbook-linked — and deleting a noisy alert improves reliability.
• Business journeys and synthetic robot-users catch what green infrastructure dashboards miss; cost is a metric with a currency symbol.

Interview Lens

Observability is the invisible section of every system design interview — invisible because most candidates skip it, which is exactly why mentioning it lands.

The probe: "Users report the app is slow. Walk me through debugging it."

The weak answer: "I'd check the logs." The strong answer walks the pillars: "Start at the SLO dashboard — is p99 actually elevated, and for which endpoints? Pull an exemplar trace from the slow bucket to see which span eats the time — gateway, service, cache miss, or database. Then the correlated logs for that requestId tell me why. Cross-check the usual suspects: cache hit rate, consumer lag, replication lag, per-instance latency for a zombie, and per-client traffic for one abusive caller."

That answer quietly demonstrates six previous articles. Interviewers notice.

Expect the follow-ups: "What would you alert on?" — user-impact SLOs, not CPU; name alert fatigue unprompted. "What's the difference between an SLO and SLA?" — internal target vs. contractual promise, with the safety gap between them. "How do you debug across ten microservices?" — correlation IDs and distributed tracing; the 2 AM story assembles from one search.

The differentiator: when given any "design X" prompt, volunteer the observability plan — "and here's what this system emits: RED per service, consumer lag on the queue, an SLO on the core journey." One sentence, instant seniority.

Real-World Engineering Lens

Make RED-per-service the default, not a decision. Baked into the service template, so every new component is born instrumented.

Middleware the correlation ID on day one. Retrofitting IDs across a live estate is archaeology; adding them at birth is one line.

Hold a monthly alert review. Every alert that fired: was it actioned? If not, fix it or delete it. Trust in the pager is a maintained asset.

Run game days. Break something in staging on purpose and check: did the evidence exist? Was the runbook right? The drill finds the gaps at 2 PM instead of 2 AM.

Put the bill on the wall. Telemetry cost, storage growth, egress — trended next to latency. Health has a financial axis.

What's Next

Pause and look at what's now on the table. Requests journey (Part 4), data persists (Blog 5), caches absorb (Part 6), messages coordinate (Part 7), storage holds (Part 8), keys distribute (Bonus G), compute multiplies (Part 9), balancers route (Bonus E), APIs promise (Part 10), trust is issued (Part 11), and now the whole machine explains itself.

Section 1 is complete. The building blocks are built.

Blog 13 starts assembling them into real systems, beginning with the classic first design: the URL shortener. But between here and there, a short run of bonus deep-dives sharpens the method itself.

Bonus A — Estimation for System Designers. The E in C.R.E.D finally gets its promised article: DAU, QPS, storage, bandwidth — the arithmetic before the architecture. Then Bonus B (how NFRs actually drive design), Bonus C (the three kinds of trade-offs), and Bonus H (polyglot persistence) — and then we build.

Question for Readers

Open your dashboards and try to answer, in five minutes: "Which users were affected between 2:00 and 2:15 last night, and by what?"

If you can't: which pillar is missing — the metric that would have alerted, the trace that would have located, or the log that would have explained?

That gap is your observability backlog, in priority order.