Rate Limiting — Algorithms and Where to Enforce Them

Why rate limiting exists

Rate limiting protects a system from being overwhelmed by request volume — whether that volume comes from a malicious actor, a misconfigured retry loop in a well-intentioned client, or simply more organic traffic than the system was provisioned for. Without it, a single misbehaving client can degrade service for every other client sharing the same backend resources.

The mechanism is simple to state — reject or delay requests beyond some threshold — but the choice of algorithm changes how that threshold behaves in practice, especially around burst traffic and boundary conditions.

Fixed window counter

The simplest approach: count requests in fixed time windows (e.g., "100 requests per minute"), reset the counter at the start of each new window.

Problem: the boundary burst. A client can send 100 requests in the last second of one window, then another 100 in the first second of the next window — 200 requests in roughly two seconds, technically compliant with "100 per minute" but violating the spirit of the limit entirely. Fixed window counters are simple to implement and reason about, but this boundary behavior is a real weakness for anything where burst protection matters.

Sliding window log

Instead of resetting at fixed boundaries, track the timestamp of every request in a rolling log, and count how many fall within the last N seconds from now, continuously — not from a fixed window start.

This eliminates the boundary-burst problem entirely, at the cost of memory: you need to store a timestamp per request (or per request within the window) rather than a single counter. At high request volumes, this can become a meaningful memory cost, though it can be mitigated with approximation techniques.

Sliding window counter

A practical middle ground: keep counters for the current and previous fixed windows, and compute a weighted estimate of the sliding window count based on how far into the current window you are. This gets most of the boundary-burst protection of a sliding log with the low memory footprint of a fixed window counter — an approximation, but a good enough one for most production use cases.

Token bucket

A bucket holds tokens, up to some maximum capacity. Tokens are added at a fixed rate (say, 10 per second). Each request consumes one token; if the bucket is empty, the request is rejected or queued.

The key property: token bucket explicitly allows bursts, up to the bucket's capacity, as long as the average rate over time stays within the refill rate. This is a deliberate design choice, not a flaw — many real workloads are naturally bursty (a client that's been idle for a while and then needs to catch up), and token bucket accommodates that burst as long as it doesn't exceed the bucket size, while still enforcing a hard average-rate ceiling over time.

Leaky bucket

Conceptually the inverse of token bucket: incoming requests fill a bucket (a queue), and the bucket "leaks" — processes requests — at a fixed rate. If the bucket is full, new requests are dropped.

The key property: leaky bucket smooths bursty input into a steady output rate, rather than allowing bursts through. This suits scenarios where the downstream system genuinely cannot handle bursts at all and needs a constant processing rate, at the cost of added latency for requests that arrive during a burst and have to wait their turn in the bucket/queue.

Choosing between them

• Need simple, cheap, "good enough" limiting and don't care about boundary bursts → fixed window.
• Need accurate limiting without boundary bursts, and can afford a bit more memory → sliding window counter (the practical default for most production rate limiters).
• Need to explicitly allow legitimate bursts up to a cap while enforcing a long-term average → token bucket.
• Need to force a strictly steady output rate regardless of input burstiness → leaky bucket.

Where in the stack to enforce limits

At the API gateway / edge, before requests reach any backend service: this protects every downstream service uniformly, is the right place for account-level or IP-level limits ("this API key gets 1,000 requests/minute total"), and stops abusive traffic before it consumes any backend resources at all.

At the individual service, for resource-specific limits: a search service might need a tighter limit than a static content service, because search queries are far more expensive to serve. Edge-level limiting alone can't express "this specific expensive endpoint needs a lower limit than the rest of the API," so service-level limiting fills that gap.

At the database or downstream dependency, as a last line of defense: even with limiting at the edge and per-service, a connection pool limit or query concurrency cap on the database protects against cases where the upstream limits were miscalculated or a legitimate-but-expensive query pattern slips through.

In practice, production systems commonly layer more than one of these — coarse-grained protection at the edge, finer-grained protection close to the specific expensive resource — rather than relying on a single limiting layer to catch every case.

The takeaway

Rate limiting algorithm choice is really a question of how precisely you need to control burst behavior versus how much memory/complexity you're willing to spend to get that precision. Get the algorithm right for your burst tolerance, and place enforcement at more than one layer so a single miscalibrated limit doesn't become a single point of failure for the whole system's protection.

This opens the Architecture Decision Deep Dives phase of the series — next up: designing for availability, and why redundancy alone doesn't guarantee it without a clear failover strategy behind it.