Load Balancing Algorithms

A load balancer does not add a single unit of capacity. It only decides where the pressure lands.

Part 9 built the fleet — twenty stateless, interchangeable instances — and ended on the obvious question: when a request arrives and twenty healthy instances could serve it, who decides which one gets it?

Part 4 introduced the load balancer and left the how for later. Bonus G promised consistent hashing. All debts come due today.

The Simple Mental Model: The Traffic Police at a Busy Junction

A load balancer is the traffic police at a busy junction. Servers are lanes. Requests are cars.

If all lanes are open and equal, send cars evenly — an easy shift. If one lane is blocked, stop sending cars there — that's what health checks are for. If one lane is moving slowly, send fewer cars there — this is the harder, more important skill, because a slow lane silently ruins more journeys than a closed one: closed lanes get avoided, slow lanes keep receiving traffic. If one road is closer to the destination, some cars should prefer it — that's locality.

And the different algorithms are simply different traffic rules the officer can enforce.

One thing the officer cannot do: add lanes. Load balancing distributes capacity; scaling (Part 9) creates it. Confusing the two is how teams end up with excellent routing to an exhausted fleet.

Where Load Balancers Sit

"The load balancer" is really a layer cake of routing decisions: DNS / global routing decides which region serves this user (the global tier — Bonus D's territory for internals). CDN / edge routing decides which edge node answers (Parts 4 and 6). The external load balancer is the classic box in front of your web/API fleet. Internal load balancers sit between services, once Part 9's microservices start calling each other. Client-side load balancing skips the middlebox entirely — the caller holds the instance list, via service discovery, and picks one itself, common for internal service-to-service traffic. The reverse proxy / API gateway handles routing plus policy at the front door (Part 10, next).

And a cousin, not a sibling: Part 7's brokers. A load balancer pushes requests at instances; a queue lets workers pull work when ready. Different pattern, same fairness goal.

Server-side vs client-side in one line: server-side puts a decision-maker in the path; client-side teaches every caller to decide. The algorithms below apply to both. Global vs regional in one more: the global tier (DNS/anycast) picks a region for the user; the regional tier picks an instance for the request. Two junctions, two officers.

Layer 4 vs Layer 7 Load Balancing

Two depths at which the officer can read the traffic.

Layer 4 — transport level. Sees IPs, ports, connections. Routes fast, asks no questions. Cannot see HTTP paths, headers, cookies, or bodies — to L4, every request is an identical envelope.

Layer 7 — application level. Opens the envelope. Understands HTTP, so it can route on meaning:

/api/orders            → order service
/api/payments          → payment service
Header: X-Beta: true   → canary version (Part 9's canary, implemented)

L7 is what makes path-based service routing, canary releases, and auth/rate-limit integration possible — at the cost of more processing per request.

The practical split: L4 for raw speed and TCP-level traffic; L7 wherever routing needs to understand the request. Modern front doors are usually L7 — which is precisely the door into API gateways, Part 10's subject.

Round Robin: Simple, Fair, and Often Good Enough

The default rule: one request each, in a circle.

Req 1 → A    Req 2 → B    Req 3 → C
Req 4 → A    Req 5 → B    ...

Works beautifully when instances are identical and requests cost roughly the same — a uniform fleet serving uniform traffic.

Fails when either assumption breaks: request 4 is a 30-second report while requests 5–9 are 20 ms lookups, and server A quietly drowns while the circle keeps turning. Instance C is half the size of A and B, so equal share means unequal suffering. Server B turns slow-but-alive, and round robin keeps feeding it one request in three, forever, unless health and outlier logic (coming below) intervene.

Round robin distributes count, not cost. Know which one your traffic needs.

Least Connections: When Requests Aren't Equal

The rule: route each new request to the instance with the fewest active connections. Active connections are a live proxy for "how busy" — a server stuck on slow requests accumulates connections and naturally stops receiving new ones.

Reach for it when request durations vary: WebSockets, file uploads (Part 8's multipart friends), long reports mixed with quick lookups.

Its blind spot: connection count isn't load. Five cheap connections can outweigh two CPU-melting ones. The signal is better than round robin's, not perfect.

Weighted Algorithms: Not All Servers Are Equal

When instances differ by design, tell the officer:

Weighted round robin and weighted least connections are the same rules, capacity-adjusted. Also handy for easing a new node in gently — low weight, then raise.

The danger is staleness: weights set by hand, then autoscaling replaces the fleet, and yesterday's weights route today's traffic. Stale weights are a self-inflicted hotspot — Bonus G's jam, configured on purpose.

Random and Power of Two Choices

Random sounds lazy and is surprisingly decent at scale — over enough requests, randomness approximates fairness. Its flaw: unlucky streaks pile requests on one instance with nobody noticing.

Power of two choices is the elegant upgrade: pick two healthy instances at random, then send the request to the less loaded of the two. That single comparison eliminates almost all of random's unlucky pileups — near-least-connections quality without global coordination, which is why the idea shows up across large distributed systems. Cheap, scalable, quietly brilliant.

Sticky Sessions and Session Affinity

Sometimes the rule is: same client, same instance, implemented via a cookie or by hashing the client IP.

Legitimate uses: genuinely long-lived interactions, and the brief, deliberate pinning Bonus I used for read-your-writes and monotonic reads — a few seconds of affinity as a consistency tool.

The trap — and Part 9 already drew the line — is stickiness used permanently, to hide local state. If sessions live in one instance's memory and stickiness is what keeps users functional, you haven't balanced load; you've assigned each user a pet server. Uneven load, painful deploys, sessions dying with instances.

Same mechanism, two verdicts: short-term affinity is a tool; permanent affinity is a scaling smell.

Consistent Hashing: Keeping Movement Small

Bonus G left a debt: naive hash(key) % N placement means that when N changes, almost every key gets a new home. For stateless web requests, who cares — any instance serves any request. But when the destination matters — a cache cluster where each node holds specific keys — mass movement is a catastrophe: add one cache node and nearly every key misses at once. Part 6 taught you what a fleet-wide cold cache does next.

Consistent hashing shrinks the movement. Picture a ring:

Ring (clockwise):
  0 ──k1──● N1 ──k2──k3──● N2 ──k4──● N3 ──k5──→ (wraps to 0)

Rule: every key walks clockwise; the first node it meets owns it.

Add N4 between N2 and N3:

  0 ──k1──● N1 ──k2──k3──● N2 ──k4──● N4 ────● N3 ──k5──→

Only k4 changed owners. k1, k2, k3, k5 never moved.

Nodes and keys are both hashed onto the same ring; adding or removing a node only reassigns the keys in its immediate neighborhood — roughly 1/N of them, instead of nearly all.

Two footnotes that matter. Virtual nodes: each physical node appears at many ring positions, smoothing out unlucky gaps in distribution. And the Bonus G caveat, restated because it's the most common misunderstanding: consistent hashing minimizes movement. It does not split a hot key. One celebrity key still maps to exactly one node — the ring just decides which node gets to suffer. Hot keys need Bonus G's medicine (caching, salting, splitting the entity), not a better ring.

Where you'll meet it: distributed cache clusters, sharded stores, and anywhere key movement is expensive — which is why Bonus G kept pointing here.

Health Checks, Outlier Detection, and Connection Draining

Algorithms decide where. This trio decides whether.

Health checks — the officer walking the lanes. An instance that stops answering gets pulled from rotation. The subtlety: check real readiness (can this instance actually serve?), not just "process alive." A process can be running and useless. And one senior-level trap: readiness should reflect this instance's ability, not the mood of shared dependencies. Wire the database into every health check, and one DB blip makes every instance "unhealthy" simultaneously, and the load balancer helpfully ejects the entire fleet.

Outlier detection — the harder problem from the mental model: the zombie server. Alive, passing health checks, and answering ten times slower than its siblings. Dead servers get removed; zombies keep receiving one-Nth of your users. Outlier detection watches per-instance latency and error rates and ejects the statistical stragglers before they die honestly.

Connection draining — the graceful exit. When an instance is being removed — deploy, scale-in, shutdown — stop sending it new requests, but let in-flight ones finish. This is the mechanism under Part 9's rolling and canary deployments: without draining, every deploy and every autoscale-in quietly kills whatever requests were mid-flight.

Retries Can Make Load Balancing Worse

A failed request, retried against a different instance, sounds sensible. It's also dangerous.

Picture a partial failure: two of ten instances are struggling, timing out 20% of requests. The load balancer retries each failure up to three times. Congratulations: at the exact moment capacity dropped, traffic grew — the retries stack on top of the original load, hammering the eight healthy instances and often tipping them over. That's a retry storm: the outage amplifier built from good intentions.

The disciplines: limits and budgets (retries as a bounded fraction of traffic, never unbounded per-request loops), backoff with jitter (Part 7's exponential-backoff lesson, applied at the balancer), and idempotency awareness (retrying a read is free; retrying a payment needs Part 7's effectively-once thinking — Part 21 goes deep).

The full defensive kit — retry patterns, circuit breakers, bulkheads — is Part 22's subject. For now: a retry is a bet that the system has spare capacity. During failures, it usually doesn't.

Load Balancing and Autoscaling Must Work Together

Part 9's autoscaler and this article's balancer are a duet, and they can absolutely sing out of tune. Discovery: new instances must join rotation automatically; scaled-out capacity nobody routes to is decorative. Warm-up: a fresh instance has cold in-process caches (Part 6) and cold connection pools; full traffic on second one is a hazing ritual — ease it in with weights or slow-start. Draining on scale-in: the autoscaler removing instances without draining is a machine for killing in-flight requests on a schedule. And the Part 9 reminder, once more: the pair can scale and route compute perfectly and still fix nothing — if the bottleneck is the database, a shard, a cache, or a dependency, flawless balancing just delivers the stampede more evenly.

When Simple Is Enough — and When It Isn't

Uniform instances, uniform requests, solid health checks → round robin, and sleep well. Variable request durations → least connections. Mixed instance sizes or gentle ramp-ins → weighted. Huge fleets, low coordination → power of two choices. Destination-matters keys (caches, shards) → consistent hashing.

In C.R.E.D terms: Clarify the traffic shape and where state lives. Requirements — fairness, affinity, locality, failure behavior. Estimate request-cost variance and instance heterogeneity. Then Design the rule, plus the health, outlier, draining, and retry policy around it.

A good architect does not ask only, "Do we need a load balancer?" A good architect asks, "What signal should decide where each request goes, and what happens when an instance becomes slow, unhealthy, overloaded, or removed?"

Common Mistakes Engineers Make with Load Balancing

Assuming round robin is always enough — it distributes count, not cost. Skipping health checks — routing to the dead. Health checks that only test "process alive" — routing to the useless. No outlier detection — routing to the zombie, the slow instance that never dies and never stops hurting users. Sticky sessions to hide local state — Part 9's trap, restated: a routing feature can't fix a state-placement mistake. Ignoring connection draining — every deploy executes a few hundred in-flight requests. Unbounded retries — the storm: less capacity, more traffic, same moment. Assuming consistent hashing fixes hot keys — it minimizes movement, it doesn't split celebrities (Bonus G). Ignoring capacity differences — equal traffic to unequal machines is a configured hotspot. Uncoordinated autoscaling — undiscovered new instances, undrained removed ones, unwarmed cold ones. No per-instance metrics — fleet averages hide zombies exactly the way Bonus G's averages hid hotspots. Expecting the balancer to fix downstream bottlenecks — it decides where pressure goes, it cannot make pressure disappear.

From My Journey: An Architectural Lesson

In many teams, load balancing is considered solved the day traffic is spread across multiple servers. A balancer in front, round robin behind it, green checkmarks all around.

And for a while, it is solved — as long as every server is equal and every server is healthy.

Production has other plans. Some requests take thirty seconds while others take twenty milliseconds. Some servers turn slow without ever dying — passing every health check while quietly ruining every third user's afternoon. Some instances arrive newer, colder, or smaller than their siblings. Some users get pinned to one machine. And during partial failures, well-meaning retries multiply the traffic at the precise moment capacity shrinks.

The realisation: the algorithm was never the whole job. Distributing requests is the easy half. The real work is routing around weakness — the slow lane, the zombie, the cold newcomer — without creating new hotspots in the process: no stale weights, no permanent stickiness, no retry storms, no celebrity keys handed to a single unlucky node.

Simple rules are easy to operate, and that simplicity is worth defending. But when workloads are uneven, instances differ, sessions matter, or failures are partial, the advanced behavior stops being optional.

The lesson that stuck: load balancing is not only about distributing requests. It is about routing around weakness while avoiding new hotspots.

A load balancer does not create capacity. It decides where pressure goes.

Key Takeaways

• Load balancing distributes capacity; scaling creates it — the officer cannot add lanes.
• Balancing happens in layers: DNS picks the region, the edge picks the node, the balancer picks the instance, and services pick each other.
• L4 routes envelopes fast; L7 opens them — path, header, and canary routing live at Layer 7.
• Round robin distributes count; least connections approximates cost; weights encode capacity; power of two choices scales gracefully.
• Short-term affinity is a consistency tool (Bonus I); permanent stickiness is a state-management smell (Part 9).
• Consistent hashing minimizes key movement when nodes change — and still cannot split a hot key (Bonus G).
• Slow instances hurt more than dead ones: readiness checks, outlier detection, and connection draining are the algorithm's bodyguards.
• Retries are a bet on spare capacity — during failures, budget them or they become the outage.
• Balancer and autoscaler must be choreographed: discover, warm up, drain — and neither fixes a downstream bottleneck.

Interview Lens

Load balancing questions test whether you've operated a fleet or just drawn the rectangle labeled "LB."

The setup: "How does traffic reach your instances?" The weak answer: "Round robin." The strong answer names the rule and its bodyguards: "L7 balancer, least-connections since our request durations vary; readiness checks that don't touch shared dependencies; outlier ejection on per-instance p99; connection draining wired into deploys and scale-in; retries budgeted with jittered backoff, idempotent requests only."

Expect the probes: "What if an instance gets slow but doesn't die?" — the zombie question; answering "outlier detection on per-instance latency" is the differentiator, answering "health checks" is the trap. "You add a node to the cache cluster — what moves?" — consistent hashing, ~1/N of keys, neighbors only, then volunteer the caveat that it still can't split a hot key. "Your retry policy?" — budgets, backoff, idempotency, bonus points for saying "retries amplify partial failures" unprompted. "Sticky sessions — yes or no?" — "Briefly, for read-your-writes; never as a home for session state."

Real-World Engineering Lens

Put per-instance p99 and error rate on the dashboard. Fleet averages are where zombies hide — the same lesson Bonus G taught about shards.

Decouple readiness from shared dependencies. One DB blip should not convince the balancer that all twenty instances died simultaneously.

Wire draining into every removal path — deploys, autoscaling, manual restarts. It's configuration once, or lost requests forever.

Give retries a budget and jitter. And mark which routes are safe to retry at all.

Start with round robin and honest health checks. Upgrade the algorithm when per-instance metrics show the need, not because a fancier name exists.

What's Next

Layer 7 opened the envelope — and the moment routing understands requests, a bigger question appears: who shapes them? Authentication, rate limits, versioning, one front door for a fleet of services.

Blog 10 — API Management: REST, gRPC, GraphQL, and API Gateways. The main series continues at the front door of the system.

Question for Readers

Tonight, one of your instances turns slow — not dead, just ten times slower than its siblings, passing every health check.

What removes it from rotation: a system, or a human reading complaints tomorrow morning?

If the honest answer is "a human," you've found this week's gap.