Scaling Compute

Traffic doubled. So we doubled the servers. Latency didn't move.

It's the most common scaling story in the industry. The dashboard turns red. The autoscaler kicks in. Ten instances become twenty-five. The cloud bill responds immediately. The users don't — the p99 is exactly where it was.

Because the bottleneck was never the application servers. It was a slow query, or a cold cache, or one hot shard quietly absorbing half the traffic while twenty-four fresh instances lined up to wait on it.

The last four articles built the supporting cast: caches that absorb reads (Part 6), brokers that carry work (Part 7), storage that holds the bytes (Part 8), keys that spread the pressure (Bonus G). Today, the layer everyone reaches for first: the compute layer — the machines, containers, services, and functions that actually run your code. And the discipline of scaling it on purpose, instead of by reflex.

What "Compute" Means in System Design

A quick separation, because these three blur together: data is the information itself (Blog 5), storage is where bytes rest (Part 8), and compute is where code executes — the CPU cycles and memory that turn a request into a response, a message into an action, an upload into a thumbnail.

Compute is the layer that does the work. Everything else feeds it or catches its output. And compute has one property that makes it special: it's the easiest layer to multiply, which is exactly why it's the layer teams multiply when they shouldn't.

We'll earn that warning by the end. First, the shapes compute comes in.

The Simple Mental Model: Kitchen Counters and Cooking Stations

One restaurant kitchen, five ways to run it.

A monolith is one large kitchen counter. Every dish — starters, mains, desserts, billing — is prepared in one place, by one team, from one setup. Simple. Fast to start. Everyone can see everything.

A modular monolith is the same kitchen with clear zones. Chopping here, cooking there, plating there, billing at the end. Still one kitchen, one door, one rent — but now the zones have boundaries, and the chaos has lanes.

Microservices are separate cooking stations. The biryani station, the tandoor, the dessert counter — each with its own team, its own tools, its own schedule. Each can grow, change, or catch fire independently.

Containers are standard food trucks. The same cooking setup, packaged so it runs identically in any parking lot — your laptop, staging, production, another region.

Serverless is hiring a temporary cook only when an order arrives. No kitchen to maintain at all. A cook materializes, makes the dish, vanishes. You pay per dish.

The lessons hiding in the metaphor: scaling one kitchen is simple, until it gets crowded. Splitting into stations lets each scale independently — the tandoor can triple without touching desserts. Too many stations creates coordination overhead — now waiters run between stations, and one order touches five teams. Food trucks make deployment repeatable — they don't make the recipes better. The temporary cook frees you from kitchen management — but works only under the agency's rules.

Everything in this article is those five sentences, expanded.

Vertical Scaling vs Horizontal Scaling

Two ways to get more compute. Vertical: a bigger machine. Horizontal: more machines.

Vertical scaling is a fine first chapter — Part 2's "simplicity wins" applies. But it always ends the same way: at the top of the price list, on a single machine, with everything riding on it.

Horizontal scaling is where real systems live. And it has one entry requirement.

Stateless Compute: The Secret Behind Easy Scaling

Blog 5 planted this seed: stateless systems are easy to scale, stateful ones preserve value but resist copying. Time to cash it in.

A stateless service instance holds no user, session, or business state of its own. Which means: any request can go to any instance.

That sentence is the entire foundation of horizontal scaling. If any instance can serve any request, then instances are interchangeable — you can add ten, kill three, and replace one mid-request, and nobody notices. The moment an instance holds something unique, the spell breaks.

Where does the state go? Somewhere shared, durable, and outside the fleet: the database, as the source of truth (Blog 5); a distributed cache for sessions and hot state (Part 6, Layer 5); object storage for files and blobs (Part 8); a message broker for in-flight work (Part 7); or an external session store when sessions deserve their own home.

The Three Classic Spell-Breakers

Local memory. Part 6 already drew this failure — the 10-servers-10-truths problem. A user's session in Server A's memory means their next request must hit Server A. Congratulations: your fleet is now a collection of tiny, mandatory monoliths.

Local disk. Part 8 named block storage "the disk your server thinks it owns" — and a deploy or instance replacement proves the "thinks." Files written locally die with the instance. Uploads belong in object storage; the instance is a hotel room, not a home.

Sticky sessions. Pinning users to instances so their local state keeps working. It functions, and it quietly forfeits everything horizontal scaling promised: uneven load, painful deploys, lost sessions on every instance death.

One important nuance so we stay consistent with Bonus I: sticky routing used briefly and deliberately — pinning a user for a few seconds after their own write, for read-your-writes consistency — is a tool. Sticky sessions as a permanent strategy for where state lives is a trap. Same mechanism, opposite reasons.

The rule: instances should be cattle, not pets — replaceable, identical, and unmourned. Anything worth keeping doesn't live on them.

Monolith, Modular Monolith, and Microservices

Now the question that starts more architecture arguments than any other: how many deployables?

The Monolith

One application, one codebase, one deploy. Simpler development, simpler debugging — one stack trace, one log, one timeline. Excellent for small teams and early systems. Scales horizontally just fine — a stateless monolith behind a load balancer is a perfectly legitimate architecture, and a criminally underrated one. The real risk isn't scale. It's coupling over time — everything can touch everything, and eventually does.

The Modular Monolith

Same single deployable, but with enforced internal boundaries. Orders code cannot casually reach into billing internals; it goes through billing's interface. You get all the operational simplicity of a monolith, plus the discipline of services, without the network in between. Often the best step before microservices — and frequently the best place to stop. Its quiet superpower: the boundaries you draw here are rehearsals for service boundaries later, drawn while mistakes are still cheap refactors instead of distributed migrations.

Microservices

Independently deployable services, each owning its slice. What you genuinely gain: independent scaling (the tandoor triples, desserts stay put), independent deploys (team A ships without waiting for team B's release train), team ownership (clear responsibility per station), and technology flexibility (the right tool per service, use sparingly).

What you now own — every word of it: every in-process function call that crossed a boundary becomes a network call, which means Part 7's whole world arrives uninvited — latency, partial failures, retries, timeouts, delayed errors. Data gets split across services, and "just join the tables" stops being a sentence. And operationally you've signed up for the full kit: per-service ownership, monitoring, logs, tracing, deployment pipelines, and incident debugging across a distributed timeline — Part 7's four-log-files problem, now as your permanent org structure.

Part 2 warned that many teams adopt microservices too early. Now you can see the mechanism: they pay the distributed-systems tax before they have the traffic, or the team count, that the tax buys anything for.

The honest framing: microservices are not a badge of maturity. They are a trade-off — organizational independence purchased with distributed complexity. Split when the coordination pain of one deployable exceeds the operational pain of many. Not before.

Containers: Packaging Compute

Whatever shape your services take, they have to run somewhere. Containers changed how.

A container image is a frozen, complete package: your application plus every dependency it needs — runtime, libraries, configuration — sealed together. The container is that image running as an isolated process.

The promise, and it's a big one: the same package runs identically everywhere. Your laptop, CI, staging, production, another cloud. "Works on my machine" stops being a punchline, because your machine ships with the code.

The one-liner: a VM pretends to be a computer; a container pretends to be alone.

And the food-truck caveat from the mental model: containers make deployment repeatable and dependencies portable. They do not make the architecture better. A tangled monolith in a container is a tangled monolith with better packaging.

Orchestration: Who Keeps the Containers Alive?

Ten containers, you can babysit. A thousand, across a hundred machines, with deploys and failures happening hourly? That's a full-time job, and it's been automated.

Container orchestration is the system that runs your containers so humans don't have to. Conceptually, it does eight jobs: scheduling (deciding which machine each container runs on), self-healing (a container dies, a replacement starts automatically at 3 AM without waking anyone), scaling replicas ("run 12 of these" becomes 12, and stays 12), rolling updates (replacing old versions gradually), service discovery (instances come and go, callers still find them), health checks (instances that stop answering get pulled and replaced), resource limits (fencing each container's CPU/memory so one can't starve its neighbors — hold this, it returns as noisy neighbor), and config and secrets (settings and credentials delivered to containers without baking them into images).

Kubernetes is the common platform for all of this — an enormous topic we are deliberately not turning into a tutorial. For system design, the concept list above is what matters: orchestration turns a pile of machines into a self-repairing fleet.

One recurring trap, named early: Kubernetes is an operations answer, not an architecture answer. It keeps whatever you designed alive. Including the mistakes.

Serverless: Compute Without Owning the Server

The far end of the spectrum: no fleet at all. You write a function. The platform runs it when triggered — an HTTP request, a queue message (Part 7!), a file landing in object storage (Part 8!), a schedule. It scales from zero to a thousand parallel executions and back to zero. You pay per execution.

Where serverless genuinely shines. Event-driven jobs (a message arrives, a function handles it), lightweight APIs (small, spiky, stateless endpoints), scheduled tasks (the nightly cleanup, the hourly report), image/file processing (upload lands in the bucket, function makes the thumbnail — Part 8's pipeline, no fleet required), glue logic (the small translator between two systems), and spiky, unpredictable workloads where a permanently-running fleet would idle at 3% utilization.

The bill, itemized. Cold starts — after idle, the first invocation must spin up a runtime, and that latency lands on a real user's request. Execution limits — functions have time and resource ceilings, long-running work doesn't fit. Vendor and platform constraints — you code to the platform's shapes, runtimes, and triggers. Harder local debugging — the production environment isn't something you can fully run on a laptop. Observability complexity — thousands of short-lived executions make Part 7's correlation-ID discipline mandatory, not optional. Cost surprises at volume — per-execution pricing is beautiful at low traffic and can quietly cross the "a fleet would've been cheaper" line at sustained high traffic. And poor fit for long-running, stateful workloads — the temporary cook cannot tend a stew for six hours.

The framing to keep: serverless removes server management, not architecture responsibility. State placement, idempotency, failure handling, cost modeling — still your job. The only thing that vanished is the fleet.

Autoscaling: Scaling on the Right Signal

Autoscaling sounds like one feature. It's actually a design question: what signal tells the truth about this workload's load?

The row that burns the most teams is the second one. Worker fleets are often I/O-bound — waiting on databases, APIs, storage. Waiting barely uses CPU. So under a massive backlog: queue depth exploding, consumers drowning… and CPU sitting at 15%. A CPU-based autoscaler looks at 15% and does the only thing it knows: it scales the drowning fleet down.

Part 7 gave workers their vital sign — consumer lag. That's the scaling signal too. Scale workers on the backlog they exist to drain, not on the CPU they barely use.

The general law: scale on the metric that measures unmet demand for that workload. The wrong metric produces waste in one direction and outages in the other, and both look like "we have autoscaling" on a slide.

One more physics note: autoscaling has lag — detect, decide, boot. A sharp spike outruns it. Which is one more quiet argument for Part 7's brokers: the queue absorbs the burst while the autoscaler catches up.

Scaling the Wrong Layer

Now the section this article exists for. Adding more application servers does not fix: slow database queries (twenty-five instances now wait on the same slow query instead of ten), a bad cache hit ratio (more misses per second, delivered faster — Part 6), a hot shard (Bonus G's jammed lane, now with more cars honking at it), a slow external dependency (you've scaled your ability to wait), a poor object-storage access pattern (Part 8's prefix hotspot, visited more often), a synchronous chain of downstream calls (Part 7's coupled chain, multiplied), a single-threaded bottleneck (one lane inside the code that no fleet size widens), or one overloaded tenant or key (the whale doesn't care how many servers watch it eat).

And here's the cruelest version, with arithmetic:

Your database accepts 300 connections.
Ten app instances × 20-connection pools = 200. Comfortable.

Traffic spikes. The autoscaler "helps": ten instances become twenty-five.
25 × 20 = 500 connection attempts against a 300-connection database.

The scaling event is the outage. You didn't fix the bottleneck — you organized a stampede at it.

That's the pattern to internalize: when compute isn't the constraint, adding compute doesn't just fail to help. It concentrates more pressure on whatever actually is the constraint.

Find the bottleneck first. Scale that layer — with Part 6's caches, Part 7's queues, Part 8's storage patterns, Bonus G's keys. Compute gets scaled when compute is genuinely the thing that's full.

Shipping Without Stopping: Deployment Strategies

A fleet also changes how you release. Three conceptual strategies: rolling deployment (replace instances gradually, a few at a time — the fleet serves traffic throughout; remember Part 6, each restart wipes in-process caches, so roll slowly enough for caches to rewarm), blue-green (run the new version alongside the old, switch traffic over in one move, keep the old version warm for instant rollback), and canary release (send a small slice of real traffic — 1%, then 5% — to the new version, watch the metrics, then widen).

Different speeds, same underlying goal: controlling the blast radius — how many users a bad release can hurt before you notice. That number is a design decision, and "all of them, instantly" is the default if you don't make it.

What Breaks at Scale

Cascading failures. One slow dependency → its callers' threads pile up waiting → they stop answering → their callers pile up. Slowness travels upstream like a wave. (The defenses — timeouts, circuit breakers, bulkheads — are Part 22's whole subject.)

Noisy neighbor. One container without resource limits eats the host; its innocent co-tenants starve. Bonus G's whale tenant, compute edition.

Resource exhaustion. Memory leaks, file handles, thread pools — anything unbounded eventually meets its bound in production.

Connection pool exhaustion. The 25 × 20 > 300 arithmetic above. Scaling one layer can DDoS the layer below it.

Autoscaling lag. The spike arrives in seconds; the new instances in minutes. Queues buy the difference, or users pay it.

Cold starts. Serverless idle-wakeup latency, landing on real requests.

Uneven traffic distribution. A fleet only helps if requests actually spread across it — which is the very next article.

Dependency explosion. Service A calls B calls C calls D; nobody can draw the graph anymore; every incident is a scavenger hunt.

Deployment blast radius. No canary, no gradual rollout — one bad deploy, every user, simultaneously.

Observability gaps. Ten services split before tracing existed: Part 7's distributed timeline with no correlation IDs. Incidents become archaeology.

Cost explosion. Over-scaling doesn't page anyone. It just invoices them. The outage-shaped problem nobody graphs until finance does.

Common Mistakes Engineers Make with Compute Scaling

Going microservices too early — paying the distributed tax before traffic or team size justifies it. Scaling app servers when the database is the bottleneck — the reflex this article exists to break. Session state in local memory — every instance becomes a pet, every deploy a small betrayal of logged-in users. Persistent files on local disk — Part 8 gave blobs a home, instance disks aren't it. Ignoring connection pool limits — fleet size × pool size must fit the database, do the multiplication before the autoscaler does. Autoscaling only on CPU — honest for APIs, a lie for I/O-bound workers. Ignoring queue depth for workers — the one metric that measures their actual job. Treating containers as architecture — they're packaging, a mess in a container is a portable mess. Treating Kubernetes as a silver bullet — it keeps your design alive, including the flaws. Serverless for long-running stateful work — the temporary cook, asked to tend a six-hour stew. No defined service ownership — an unowned service is Part 7's unowned queue, an outage with no adult assigned. Splitting services before observability exists — distribute the system only after you can see into it (Part 12 is coming for exactly this reason).

Applying C.R.E.D to Compute

Clarify — What work is actually being scaled? Request-serving, background processing, scheduled jobs? Where does its state live today?

Requirements — Latency targets per workload, deploy independence needs, team ownership boundaries, tolerance for cold starts.

Estimate — Traffic per workload, read/write and CPU-vs-I/O profile, connection math (fleet × pool vs database limits), burst size vs autoscaler lag.

Design — The execution model per workload (monolith zone, service, container fleet, function), the scaling signal for each, the deploy strategy, and — always — where the state moved to.

A good architect does not ask only, "How many servers do we need?" A good architect asks, "What work are we scaling, how stateful is it, how independently does it change, and what operational complexity are we willing to own?"

From My Journey: An Architectural Lesson

In many teams I've watched hit their first real scaling wall, the response is the same — and honestly, it's an understandable one: add more application servers. It's the most visible layer. It's the layer with a slider.

Sometimes it even works. When the app is stateless and compute is genuinely the constraint, more instances are exactly the medicine.

But often the real bottleneck is somewhere else entirely — database load, a poor cache hit ratio, one hot shard, a slow external service, a synchronous chain of downstream calls. And then the new servers don't relieve the pressure. They forward it. More instances waiting on the same query. More connections crowding the same database. More retries hammering the same dependency.

The realisation that reframes everything: scaling compute is not the same as scaling the system. Compute is one layer. It has to be scaled in conversation with state, data, storage, queues, and dependencies — or it just buys the failure a bigger audience.

And every model on the menu turned out to be a trade. Monoliths: simpler, until crowded. Microservices: independent, at the price of distributed complexity. Containers: repeatable packaging, not better architecture. Serverless: no servers to manage, new constraints to obey.

The lesson that stuck: scaling compute is not about multiplying servers. It is about placing work in the right execution model, and moving state out of the places that need to scale.

More servers do not fix the wrong bottleneck. They only help the wrong bottleneck fail faster.

Key Takeaways

• Compute is where code executes — the easiest layer to multiply, and therefore the most misleadingly scaled.
• Horizontal scaling has one entry fee: statelessness. Any request, any instance; state lives in the database, cache, object storage, or broker — never on the fleet.
• A stateless monolith behind a load balancer is a legitimate, underrated architecture; the modular monolith is a rehearsal for services with refactor-priced mistakes.
• Microservices trade coordination pain for distributed complexity — network calls, split data, and the full observability kit. A trade-off, not a badge.
• Containers make packaging repeatable; orchestration keeps fleets alive; neither improves the architecture inside.
• Serverless removes server management, not architecture responsibility — and it bills you for cold starts, constraints, and volume.
• Autoscale on the signal that measures unmet demand: requests for APIs, queue depth for workers. CPU lies about I/O-bound work.
• When compute isn't the constraint, adding compute concentrates pressure on whatever is — do the connection-pool math before the autoscaler does.
• Deployment strategy is blast-radius design: rolling, blue-green, canary.

Interview Lens

Compute scaling hides inside the most common prompt in system design interviews: "Your API is slow under load. What do you do?"

The weak answer: "Add more servers / turn on autoscaling." The strong answer refuses to scale blind: "First I find the constraint — p99 by layer: is CPU actually hot, or are instances waiting on the database, the cache hit rate, one shard, or a downstream call? If compute is genuinely full, scale horizontally — the service is stateless, sessions in Redis, files in object storage. If it's not, more instances just crowd the real bottleneck — and I'd check the connection-pool math before any scale-out."

Expect the probes: "Monolith or microservices for this?" — modular monolith until deploy contention, team count, or divergent scaling profiles force the split; saying "monolith, deliberately" with reasons reads as senior, not junior. "How do your workers scale?" — queue depth / consumer lag, never CPU, and explain the 15%-CPU-while-drowning trap; this one answer separates operators from readers. "What does stateless actually mean?" — any request to any instance, then name where each kind of state went. "How do you deploy without downtime?" — rolling/blue-green/canary, framed as blast-radius control.

The differentiator: mention the second-order effects unprompted — connection pools, autoscaler lag, cold caches after restarts. Interviewers are listening for evidence you've watched a fleet, not just drawn one.

Real-World Engineering Lens

Prove the bottleneck before scaling anything. One dashboard, p99 per layer — compute, cache, database, queue, external. Scale the layer that's actually red.

Run the statelessness audit. Search the codebase for local file writes and in-memory session maps. Every hit is a future 2 AM incident with a deploy in its stack trace.

Do the connection math in the design review. Max fleet size × pool size vs database limit. Write the number down next to the autoscaler config.

Autoscale workers on lag from day one. The CPU-based default is a quiet time bomb for I/O-bound fleets.

Set resource limits on every container. Noisy-neighbor insurance costs two lines of config.

Canary anything with users. 1% of traffic finding a bug is an alert; 100% finding it is an apology.

What's Next

We've built the fleet: stateless instances, containers, orchestration keeping them alive. Which surfaces a question we've been waving at since Part 4: when a request arrives and twenty healthy instances could serve it, who decides which one gets it?

Part 4 introduced the load balancer as a traffic controller and promised we'd return for the how. Bonus G promised consistent hashing. Both debts come due now.

Bonus E — Load Balancing Algorithms: Round Robin, Least Connections, and Consistent Hashing.

After that, the main series continues with Blog 10 — API Management: REST, gRPC, GraphQL, and API Gateways. The front door of the system: how requests are shaped, versioned, secured, and rate-limited before they ever reach the fleet.

Question for Readers

If your traffic doubled in the next ten minutes: which layer falls over first — compute, cache, database, queue, or a dependency?

And the sharper follow-up: would adding app servers help, or would it just organize a bigger stampede at the real bottleneck?

If you don't know the answer, your monitoring just told you something too.