Caching at Scale

The fastest database query is the one you never make.

In the last article, we ended with a question: if databases are so powerful, why don't we query them for everything? Let's answer it with a scene.

It's 8 PM on a Friday, and a food delivery app is showing the menu of a popular restaurant. Within one minute, 40,000 hungry people open that same menu. The menu changed twice today — should the database really answer the same question 40,000 times? Of course not. Answering the same question again and again, when the answer hasn't changed, is wasted work — and caching exists to eliminate that waste.

But here's what most engineers get wrong: they think caching is a tool. It isn't. Caching is a series of design decisions, and by the end of this article you'll see why.

What a Cache Really Is

Strip away the technology, and a cache is simply a copy of data, stored closer to where it's needed, so expensive work can be skipped. Two words in that sentence carry all the weight: copy, and closer.

In Part 5, we established a foundational idea: every system needs a source of truth. Here's the corollary: a cache is never the source of truth. A cache is a derived copy.

And the moment you copy data, you create a brand-new problem — the copy can become wrong. We have a name for that: stale data.

Every caching decision you will ever make is a negotiation between two forces:

     Speed  ←——————————→  Freshness
Serve the copy          Ask the source
(fast, maybe stale)     (slow, always true)

Hold on to that tension. It is the entire article in one picture.

Why Caching Changes Everything

Let's put rough numbers on it — typical orders of magnitude, actual figures vary by setup:

Read from application memory     ~0.1 ms
Read from Redis over network     ~1 ms
Read from database               ~10–100 ms
Heavy aggregation query          100 ms – several seconds

These match what we saw in Part 4. But latency is only half the story. The other half is load.

Suppose your cache answers 95 out of every 100 requests. The database now sees only 5. That's not a small win — that's a 20× reduction in database traffic.

A Very Indian Example

During an India cricket match, a live-score page may receive millions of identical requests per minute — millions of people asking the same question at the same time. Now suppose you cache the score with a TTL of just one second. One second of staleness. Nobody notices. But instead of millions of database reads per minute, the database sees roughly one read per second.

This is the point people miss: even a tiny TTL can absorb a massive amount of traffic. This is how systems become dramatically faster without changing their database at all. The database doesn't get faster. It simply stops being asked.

The Mistake: Drawing Cache as One Box

In Part 4, our request journey showed a single cache sitting between the application and the database — App → Cache → Database. That was a deliberate simplification to keep the journey readable, and we promised to come back and study caching properly. This is that article.

Because here's the truth about production systems: there is no single cache. Caching is a stack. By the time a request reaches your database, it may have passed six different caching opportunities:

User
 ↓
① Browser Cache            (on the user's device)
 ↓
② CDN / Edge Cache         (in the user's city)
 ↓
③ Reverse Proxy Cache      (at your front door)
 ↓
④ In-Process Cache         (inside your app server's memory)
 ↓
⑤ Distributed Cache        (Redis / Memcached cluster)
 ↓
⑥ Database Cache           (buffer pool, query cache)
 ↓
Source of Truth (disk)

One rule governs this entire stack: the closer a cache sits to the user, the faster it is — and the less control you have over it. Let's walk down the stack, layer by layer.

Layer 1: Browser Cache

The first cache isn't in your infrastructure at all — it's on your user's phone or laptop. When a browser downloads your logo, CSS, or JavaScript bundle, it can keep a copy locally. Next visit: the browser checks its local copy, and if it's still valid, there's no network request at all. Zero network. Zero server load. Instant.

You control this layer indirectly, through HTTP caching headers like Cache-Control and ETag.

What belongs here: logos, fonts, images, CSS and JavaScript bundles, anything static and public. What doesn't: personal or sensitive data (the device may be shared), and anything you might need to take back urgently.

Because here's the catch: once a browser has cached a file with a long TTL, you cannot reach into the user's device and delete it. That's why engineers use versioned filenames — app.js is risky to cache for long, but app.4f9a2c.js is safe to cache for a year. Change the content, change the name. The old copy becomes irrelevant instead of dangerous.

Layer 2: CDN / Edge Cache

We met CDNs in Part 4: servers spread across the world, keeping content close to users. A CDN is fundamentally a giant, geographically distributed cache.

What belongs here: images, videos, static assets, and increasingly, entire API responses for anonymous content, like a public product page. What doesn't: user-specific responses, unless you are extremely careful.

This layer hides one of the most dangerous caching mistakes in the industry.

The Cache-Key Leak

Suppose your CDN caches GET /api/my-orders without including who is asking in the cache key. User A requests their orders — the CDN caches the response. User B requests their orders. The CDN serves User A's orders back to them.

This isn't hypothetical — this exact class of incident, a cached response served to the wrong user because identity wasn't part of the cache key, has hit real production systems before.

The rule: if the response depends on who is asking, identity must be part of the cache key — or the response must not be edge-cached at all.

Layer 3: Reverse Proxy / Gateway Cache

Now we're inside your infrastructure. In Part 4, we saw the reverse proxy as the smart gatekeeper. One of its powers: caching entire responses at your front door, using tools like Nginx or Varnish.

Internet
   ↓
Reverse Proxy   ← cached homepage lives here
   ↓
Application Servers (never bothered)

What belongs here: anonymous, high-traffic pages — the home page, category listings, public search results. What doesn't: personalized pages, or anything that varies per user session.

Why this layer is special: unlike the browser and the CDN, you fully own this cache. You can inspect it, purge it instantly, and control the keys. That makes it the first layer where invalidation is genuinely in your hands. (We'll expand on gateways and API management in Part 10.)

Layer 4: Application In-Process Cache

This is the fastest cache that exists — a simple map, living inside your application server's own memory:

config["delivery_fee"] → 25

No network hop, no serialization, near-instant access.

What belongs here: small, hot, rarely-changing data — configuration values, feature flags, country lists, category trees. What doesn't: large datasets (you'll eat your own application's memory), or anything that must look identical across all servers at the same moment.

That second point deserves a diagram — call it the ten-servers-ten-truths problem:

         Load Balancer
       ↓       ↓       ↓
  Server A  Server B  Server C
  cache: v1 cache: v2 cache: v1

Each instance holds its own private copy. Update the data, and for a while, different servers serve different answers, depending on which one the load balancer picks. Also: every deploy or restart wipes this cache back to zero — remember that word, zero. It returns later in this article.

Layer 5: Distributed Cache

This is the layer most engineers mean when they say "cache" — Redis, Memcached: a dedicated, shared, in-memory store that all application servers talk to.

 Server A   Server B   Server C
      ↘        ↓        ↙
        Distributed Cache
               ↓
            Database

One shared copy. Every server sees the same cached value. In Part 4, we promised to study Redis-style caching deeply. Here it is.

What belongs here: user sessions, hot database entities (that restaurant menu), expensive computed results, counters and rate-limit state.

The Default Pattern: Cache-Aside

The most common way applications use a distributed cache:

READ:
App → Cache: "menu:123?"
        Hit  → return it              (fast path)
        Miss → App → Database
             → store in cache (TTL)
             → return it              (slow path, once)

WRITE:
App → Database                 (source of truth first)
App → Cache: delete "menu:123"

The next read misses, fetches fresh data, and repopulates the cache. Simple. Predictable. The 80% solution.

The Pattern Family

That last one — write-behind — quietly depends on reliable message delivery. We'll meet the machinery behind that in Part 7: Messaging Systems.

The Cost of This Layer

Two things to remember. First, it's a network hop — around a millisecond. Fast, but not free. Second, and far more important: the distributed cache becomes critical infrastructure. If it disappears, every request it was absorbing lands on your database, at once. Hold that thought — we're coming back to it.

Layer 6: Database-Level Cache

Here's a secret: even when every layer above misses, your database is still trying to cache for you. The buffer pool / page cache keeps recently read data pages in the database server's memory; query-level caching lets some engines cache results or execution plans; materialized views precompute query results, refreshed on a schedule.

Ever noticed a query is slow the first time and fast the second time? That's this layer working. And notice what a materialized view really is: a cached answer inside the database, with a refresh policy — and a refresh policy is just TTL wearing a formal shirt.

The lesson of Layer 6: even your source of truth caches internally — but this layer can't save you from network round trips or repeated application work. That's exactly why the other five layers exist.

The Stack at a Glance

Six layers. Six different owners, speeds, and failure stories. Now the real questions begin.

TTL: The Freshness Dial

Every cached item must answer one question: how long is this copy allowed to live? That's the TTL — Time To Live — and it's a dial, not a switch:

Here's the uncomfortable truth: there is no correct TTL. There is only a TTL that matches how much staleness a specific piece of data can tolerate. A restaurant menu can tolerate five minutes of invisible staleness. A product price, maybe a minute, depending on the business. Seat availability at final booking confirmation? Zero — ask the source of truth.

Which means TTL is a per-data decision, not a per-system decision. This is Part 2's lesson wearing new clothes: no perfect answers, only trade-offs that fit a context.

Cache Invalidation: The Second Hard Thing

There's a famous saying in computer science, usually attributed to Phil Karlton: "There are only two hard things in Computer Science: cache invalidation and naming things."

Why is invalidation hard? Because when data changes, someone must answer which of the six layers hold a copy, who is responsible for clearing each one, and what happens in the seconds (or hours) before every copy is updated. Three broad strategies exist.

Strategy 1: Let it expire (TTL). Do nothing. Accept bounded staleness. Simple, predictable — and for a lot of data, completely fine.

Strategy 2: Invalidate on write. When the restaurant edits its menu, actively delete or refresh the cached copies:

Update menu in DB
   ↓
Delete "menu:123" from Redis
   ↓
Purge /restaurant/123 from CDN

Precise — but now your write path must know every place a copy lives. That's coupling, and coupling has a cost.

Strategy 3: Version the key. Never delete anything. Change the address instead:

menu:123:v41  → old, ignored
menu:123:v42  → new requests go here

Hashed asset filenames (app.4f9a2c.js) are exactly this trick.

The Ownership Rule

Whatever strategy you pick, one rule is non-negotiable: every cache layer must have a named owner and a written invalidation story. A cache nobody owns is not an optimization. It's a lie generator with a delay timer.

What Not to Cache

By now, caching sounds wonderful. Let's balance the scales — knowing what not to cache is half the skill.

Fast-changing data. If data changes as often as it's read, caching adds staleness without saving work. A live auction's current bid has no business sitting in a five-minute cache.

Correctness-critical reads. A bank balance before approving a withdrawal. Seat availability at the final booking confirmation. The remaining uses of a coupon at the moment of redemption. Cache the browsing view of a seat map if you like — never cache the confirmation check. When correctness is on the line, ask the source of truth, and gladly pay the extra milliseconds.

Sensitive user-specific data, unless carefully controlled. Per-user data in shared caches demands three things: user identity in the cache key, short TTLs, and real thought about where copies land. Get it wrong, and you've built the cache-key leak from Layer 2.

Data where staleness costs more than the query. Run the simple math: the cost of a database read is milliseconds of latency; the cost of a wrong answer is refunds, support tickets, broken trust. If the second number is bigger, the cache is a bad deal.

Data with unclear ownership or invalidation rules. If nobody on the team can answer "when this data changes, what clears the cache?" — you are not caching, you are scheduling a future bug.

Slow queries you haven't understood. The most seductive misuse of caching is "this query takes four seconds, let's cache it." Sometimes that's reasonable. But if you never ask why it takes four seconds, you haven't fixed the problem — you've hidden it. And hidden problems have a way of choosing their own moment to return, which brings us to the most important section of this article.

When the Cache Becomes a Mask

This pattern repeats across the industry. A slow query hides behind a 99% cache hit rate. Dashboards look green, latency looks great, everyone moves on. The four-second query is still there. Invisible.

Then one day, a deploy restarts every app server, wiping every in-process cache back to zero — or the distributed cache restarts, or thousands of TTLs happen to expire together. And suddenly:

Warm cache:  10,000 requests → 99% hits → DB sees ~100    → fine
Cold cache:  10,000 requests → all miss → DB sees 10,000  → meltdown

Every miss triggers the four-second query. The database saturates. Requests time out. Retries pile on top of the timeouts. This failure has a name: cache stampede, also called the thundering herd.

The defenses, briefly: jittered TTLs (add randomness so keys don't expire in unison), request coalescing (on a miss, let one request rebuild the value while the others wait), cache warming (pre-fill hot keys before traffic hits, especially around deploys), and serve-stale-while-refreshing (briefly serve the old value while a fresh one is fetched).

We'll treat these failure patterns properly — alongside retries, circuit breakers, and bulkheads — in Part 22: Designing for Failure.

For now, internalize the principle: if your system is only healthy while the cache is warm, the cache is no longer an optimization. It is a load-bearing dependency, and it deserves the same design attention as your database.

Applying C.R.E.D to Caching

Caching decisions fit the same framework we've used since Part 3.

Clarify — Which reads are actually hot? What does a user experience if this data is stale?

Requirements — Define freshness tolerance per data type. Menu: minutes. Price: less. Seat at checkout: none.

Estimate — What's the read/write ratio? How large is the hot working set? What hit rate is realistic? Caching write-heavy, rarely-read data is decoration, not design. (We'll build estimation skills properly in Bonus A.)

Design — For each hot dataset, specify four things: the layer, the TTL, the invalidation path, and the failure behavior.

If you can fill those four blanks, you're not "adding a cache." You're designing one. And that's the whole shift: a good architect does not ask only, "Should we use cache?" A good architect asks, "What should be cached, where, for how long, and what happens when it becomes stale?"

From My Journey: An Architectural Lesson

In one project, certain screens simply felt slow. When we looked closer, the pattern was obvious: repeated reads were hitting the backend and the database again and again, for data that barely changed.

The first instincts were the usual ones — optimize the queries, add infrastructure, or just "put a cache in front of it and make it fast." But the real issue was hiding one level deeper. We had never clearly separated the data that needed real-time freshness from the data that could comfortably tolerate short-lived staleness.

Once we asked that question, caching stopped looking like a performance trick and became an architecture question. Where should each cache sit? Who owns it? How long should data live there? And what happens after the original data changes?

Every answer was a trade-off. Longer TTLs improved performance but increased stale-data risk. Shorter TTLs improved freshness but gave up much of the benefit. Some data belonged close to the user. Some belonged near the application. And some data, we realized, should not have been cached at all.

That experience left me with the lesson this entire article is built on: caching is not one layer. It is a layered design decision.

Key Takeaways

• A cache is a copy of data stored closer to where it's needed — it is never the source of truth.
• Caching is a stack of six layers — browser, CDN, reverse proxy, in-process, distributed, and database — not a single box.
• The closer a cache sits to the user, the faster it is and the less control you have over it.
• TTL is a per-data decision that must match each dataset's tolerance for staleness.
• Every cache layer needs a named owner and a written invalidation story.
• Some data should never be cached: correctness-critical reads, fast-changing values, and anything with unclear invalidation rules.
• High hit rates can hide broken queries — cold caches reveal them, sometimes catastrophically.
• If your system is only healthy when the cache is warm, the cache is a dependency, not an optimization. Design it like one.

Interview Lens

Here's how caching shows up in interviews — and how to stand out.

The weak answer: "We'll add Redis." The strong answer names five things in one sentence: "I'd cache the restaurant menu in a distributed cache with a 5-minute TTL, invalidated on menu updates; if the cache is cold, request coalescing protects the database." Data. Layer. TTL. Invalidation. Failure behavior.

Expect these follow-up probes: "What's your invalidation strategy?" (TTL expiry vs. invalidate-on-write vs. versioned keys), "What happens if the cache goes down?" (this is the stampede question in disguise), "How big should the cache be?" (working set size and read/write ratio, not a guess), and "What about a hot key?" (one celebrity restaurant receiving most of the traffic — we'll meet this again in feeds and fan-out, Part 16).

And one differentiator most candidates miss: mention the layers before Redis. Saying "this content should actually be cached at the CDN, not in the application" signals system-level thinking immediately.

Real-World Engineering Lens

Here's how this plays out in production work.

Measure before you cache. Know the read/write ratio and the expected hit rate first — a cache added on instinct is a guess with a TTL. Start boring: cache-aside plus Redis plus honest TTLs is the 80% solution, and exotic patterns need to earn their complexity. Write the invalidation story in the design doc — one sentence per cached dataset, what clears it and who owns it. Future-you will be grateful.

Respect deploy day. Every restart wipes in-process caches; warm critical keys, stagger rollouts, or expect a thundering herd at your database. Run the audit question — "what breaks if the cache is empty right now?" — and answer it in a design review, before production answers it for you.

(Measuring hit rates and cache health properly belongs to observability — that's Part 12.)

What's Next

We're going to make one short, deliberate detour before Part 7.

Bonus F — PACELC: Beyond CAP Theorem. In Part 2 and Blog 5, we saw CAP: when a network partition happens, you choose between consistency and availability. But here's the question CAP doesn't answer: what about when everything is working fine? PACELC completes the picture — because even a perfectly healthy system trades latency against consistency every single day. Every replica — and yes, every cache in this article — is quietly making that trade.

After that, the main series continues with Blog 7 — Messaging Systems: Queues, Streams, and Event-Driven Architecture, where we'll answer a question this article quietly raised: when data changes, how do invalidation events actually travel through a large system?

Question for Readers

Think about the system you work on today. Pick one piece of cached data — any one. Can you name its TTL, its owner, and exactly what clears it when the source changes?

If yes, you're ahead of most teams. If no — you've just found your first architecture task for the week.