Social Networks and Feeds

A normal user posts. A thousand people see it. Easy.

A celebrity posts. Fifty million people see it — in the same instant.

Those are not the same feature. They are barely the same system.

Section 2 built three systems whose scale stayed polite. Part 15 ended on a fence: cap the group at 256, because past that, a "group chat" becomes a broadcast feed, and feeds have different physics.

This is that different physics.

And it's where a debt finally comes due. Bonus G named the celebrity problem back in the caching articles and kept deferring it — "Part 16 owns feed-scale fan-out." Part 15 pointed here explicitly. Every "we'll face this later" in the series was pointing at today.

Section 3 changes the altitude. Parts 13–15 designed one system each. From here, each article designs a category — the shape shared by Twitter, Instagram, LinkedIn, TikTok, and every feed you've ever scrolled. Same template as the mini-systems, one altitude higher.

Why the Feed Is the Hardest Problem So Far

The numbers stop being polite. Every prior system had a merciful ceiling — ½ TB of URLs, 256-member groups. A feed's fan-out has no such mercy: one action can generate tens of millions of downstream writes, and the distribution is savagely uneven. This is Bonus G's hot key with the safety rails removed.

Reads and writes are both hard, and they trade against each other. Part 13 was read-heavy with trivial writes. Here, the same design decision moves cost between the read path and the write path, and picking wrong at either end melts something. Fan-out is the purest read-vs-write trade in the series.

"Correct" stops being definable. A URL redirect is right or wrong. A feed's ordering is a product opinion — chronological, ranked, algorithmic — and there's no ground truth to check against. We inherit Part 13's lesson (immutability is a gift) and lose it entirely: a feed is the opposite of immutable.

And it composes everything. Caching (6), messaging and fan-out (7), storage (8), sharding and the hot key (Bonus G), stateless compute (9), the estate's other systems (13–15) — the feed uses all of it at once. It's the series' integration exam.

A good architect does not ask only, "Can we show people what their friends posted?" A good architect asks, "Where does the write go, who pays for the read, and what happens when one account is a million times bigger than the rest?"

Step 1 — Clarify: The Questions Before the Boxes

"Design a social feed" describes ten different products. The separating questions: feed type — following-only (Twitter-classic), algorithmic (TikTok), or friends-graph (Facebook)? Ordering — reverse-chronological, or ranked/algorithmic? Follower distribution — symmetric friends (capped-ish) or asymmetric follows (celebrities with millions)? Content — text, images, video — the payload shape drives storage (Part 8). Real-time — must a post appear in followers' feeds in seconds, or is a delay fine? Scale — thousands of users, or hundreds of millions?

Our assumptions, stated (Bonus A's rule):

Feed type:        Following-based, with ranking on top
Ordering:         Ranked — recency + relevance (not strict chronological)
Follows:          ASYMMETRIC — most users have hundreds; some have MILLIONS
Content:          Text + images + video, all by reference (Part 8)
Real-time:        Seconds for most; the celebrity case may lag — by design
Scale:            100M users — the numbers stop being polite here

That third answer — asymmetric follows — is the entire article. A world where everyone has ~200 followers is easy. The celebrity is what makes this hard, and we chose to face it head-on.

Step 2 — Requirements: The Building Code

Functional: publish a post; generate each user's home feed (posts from people they follow); rank it; support likes/comments (counters at scale); deliver new posts in near-real-time.

Non-functional — the numbers that shape the boxes:

Feed read latency p99      < 200 ms   (feeds are scrolled; slowness = abandonment)
Feed freshness             seconds for normal; celebrity posts may take longer
Read/write ratio           read-DOMINANT — feeds are read far more than posted
Fan-out amplification       1 post → up to tens of millions of feed insertions
Ranking                     applied at read or precomputed — a core trade-off
Availability               99.9% reads; a missed post is recoverable, a down feed is not
Counter accuracy           likes/views: approximate-at-scale is acceptable
Retention                  posts durable long-term; feeds are rebuildable (Bonus H)

Read the fan-out line again. "Up to tens of millions of feed insertions per post" is not a number any other system in this series has produced. It is the reason the next section exists.

Step 3 — Estimate: The Numbers That Force the Design

Bonus A's method — and this time the arithmetic doesn't just inform the design, it dictates it:

Assume: 100M users, 10% post daily, avg post reaches followers

POSTS
  10M posts/day ÷ ~100K sec  ≈ 100 posts/sec average
  × 5 peak                   ≈ 500 posts/sec       ← modest!

FEED READS
  100M users × ~10 feed-opens/day = 1B reads/day
  ÷ ~100K sec                ≈ 10,000 reads/sec avg
  × 5 peak                   ≈ 50,000 reads/sec    ← the real load

FAN-OUT — where it breaks
  Average user (200 followers):
    1 post → 200 feed insertions. Fine.

  Celebrity (50M followers):
    1 post → 50,000,000 feed insertions.
    At 500 posts/sec including a few celebrities,
    write amplification can spike into the BILLIONS/sec.
    ↑ THIS is the number that no single strategy survives.

STORAGE
  Posts:  10M/day × ~1 KB text ≈ 10 GB/day (media → object storage, Part 8)
  Feeds:  if precomputed — 100M users × ~500 cached entries
          = 50 BILLION feed entries to maintain. Derived, but enormous.

The verdict lands hard: posting is cheap, reading is heavy, and fan-out is lethal. The celebrity's 50-million-insertion post is the single fact the entire architecture must bend around. Every design decision below is a response to that one number.

The Core Decision: Fan-out on Write vs Fan-out on Read

Everything in a feed system reduces to one question: when a post is created, do you push it to followers' feeds, or wait and pull it when they read?

Fan-out on Write (Push)

When you post, the system immediately writes your post into every follower's precomputed feed.

Post created
   → look up all followers
   → insert post into each follower's feed cache/store
   → when they open the app, their feed is ALREADY BUILT

Read is trivial — the feed is precomputed; opening the app is one cheap lookup (Part 6). p99 < 200 ms is easy. Write is the cost — one post equals N insertions; for a normal user, fine, for a celebrity, catastrophic: 50M writes from one tap. Storage balloons — every follower stores a copy of every post they might read (the 50-billion-entry problem).

Fan-out on Read (Pull)

When you post, you just store the post. When a follower opens their feed, the system fetches recent posts from everyone they follow and merges them, live.

Feed opened
   → look up everyone this user follows
   → fetch recent posts from each
   → merge, rank, return

Write is trivial — one post, one write; the celebrity posts as cheaply as anyone. Read is the cost — every feed open queries hundreds of followees and merges; for a user following 5,000 people, that's a heavy read, every scroll. Freshness is free — you always pull the latest, nothing to invalidate.

The Trade, Side by Side

Stare at that table and the punchline appears: push breaks on the celebrity's write; pull breaks on the heavy-follower's read. Neither wins. This is Bonus C's physics trade-off in its purest form — no budget, no cleverness makes both cheap. Which is why real systems refuse to choose.

The Hybrid: How Real Feeds Actually Work

The answer isn't push or pull. It's push for the many, pull for the few, split by exactly the thing that breaks push:

Normal user posts (hundreds of followers)
   → FAN-OUT ON WRITE: push into followers' feeds. Cheap, fast reads.

Celebrity posts (millions of followers)
   → DO NOT push to millions. Store the post once.
   → FAN-OUT ON READ: when a follower opens their feed,
     merge precomputed (from normals) + freshly-pulled celebrity posts.

So a follower's feed is assembled from two streams: the precomputed base — posts pushed by the accounts they follow who aren't celebrities (already sitting in their feed cache) — and the pulled overlay — recent posts from the handful of celebrities they follow, fetched at read time and merged in.

The celebrity's single post is now written once and read by millions — which is exactly Part 13's insight resurfacing: a read-heavy, rarely-changing object is the cache's favorite meal. One celebrity post cached and served 50 million times is trivial. Fifty million copies written into 50 million feeds is a meltdown. Same content, opposite economics — the whole design is choosing the right one per account.

Where's the celebrity threshold? A follower count above which an account flips from push to pull — say, 100K+ (a tuned number, not a law). This is Bonus C wearing work clothes: a business/tuning decision, monitored and adjusted, not a constant hardcoded once.

And notice the elegance: the hybrid spends the cache on the celebrity (few objects, massive reuse) and spends the fan-out on normal users (many objects, little reuse each). Each strategy is applied exactly where its economics win.

Deep Dive: Feed Ranking

Chronological feeds were the old world. Modern feeds rank, which changes the data problem.

Ranking needs signals: recency, engagement (likes/comments velocity), affinity (how much you interact with this author), content type, and dozens more. Computing a personalized score for every candidate post, at read time, for 50,000 reads/sec, is a latency budget nobody can afford (Bonus A).

So ranking splits into stages — the pattern behind most large feeds:

CANDIDATE GENERATION   gather the pool: pushed feed + pulled celebrity posts
                       + candidate sources (a few hundred posts)
        ↓
LIGHT RANKING          cheap scoring to trim hundreds → dozens
        ↓
HEAVY RANKING          expensive model scores the survivors (personalization)
        ↓
FEED                   the ranked page the user sees

Two architectural consequences. Precompute what you can, rank what you must — heavy ranking on a small candidate set fits the budget, heavy ranking on everything does not; this is the read-vs-write trade again, now inside ranking itself. And ranking makes feeds non-deterministic and non-immutable — the same user, refreshing twice, may see a different order. Part 13's immutability gift is fully gone, and with it, easy caching. Feed caches now carry TTLs and get rebuilt (Bonus H: derived, rebuildable, never truth).

We will not turn this into an ML article. The system-design point is the shape: staged ranking exists to keep an expensive computation inside a tight latency budget by shrinking the input at each stage.

Deep Dive: Counters at Scale — Likes and Views

A viral post gets 5 million likes in an hour. Naively, that's 5M increments to one row — Bonus G's hot key at its most vicious, a single counter melting under concurrent writes.

The requirement rescues us: the NFR said approximate-at-scale is acceptable. Nobody needs to know a viral post has exactly 4,211,109 likes versus 4,211,110. That tolerance unlocks cheaper designs. Sharded counters — split one counter into N sub-counters across shards, increment a random shard, sum them for display; the hot row becomes N warm rows (Bonus G's salting, applied to counts). Batched/async aggregation — buffer increments and flush periodically (Part 7); the display count lags by seconds, which the NFR permits. Approximate reads — cache the count with a short TTL (Part 6's cricket score, exactly): millions of reads served from cache, the true count catching up behind.

The lesson is the one this series keeps teaching: the consistency requirement, not the technology, unlocks the design. "Exact like counts" would force expensive coordination; "approximate is fine" makes it cheap. Bonus F's question — what can be wrong, for whom, for how long — answered once, pays off across the whole counter subsystem.

Failure Analysis: Where This Design Bends and Breaks

A mega-celebrity posts at peak. If they were on the push path, 50M insertions would saturate the fan-out workers and back up every other post behind them. That's why they're on the pull path — the threshold exists precisely to prevent this. The failure is designed out, not handled after.

Fan-out workers fall behind. Normal-user posts queue; feed freshness lags from seconds to minutes (Part 7's consumer lag, watched). Feeds are eventually consistent by design (Bonus I) — a post arriving 90 seconds late is invisible; the fan-out never sat on the read path. Backpressure and worker scaling (Part 9, on queue depth) drain it.

The feed cache dies. Precomputed feeds vanish, the system falls back to pull-and-rebuild, a load spike on the post stores (Part 6's cold-cache stampede, at feed scale). Defenses: rebuildable feeds (Bonus H), request coalescing, staged warming. And the honest question: can the stores survive a cold feed cache at peak? If not, it's a load-bearing dependency, designed accordingly.

A "normal" user crosses the celebrity threshold mid-growth. The account that had 80K followers yesterday has 120K today. The system must migrate them from push to pull — a threshold that only triggers on new posts, plus monitoring for accounts approaching the line (Bonus G's "watch the account approaching whale status," here, automated).

A viral post's counter melts. Sharded counters absorb it; approximate reads serve it. The hot key that would have killed a naive design is defanged by the NFR.

The ranking service degrades. Heavy ranking times out → graceful degradation to light ranking or chronological (Part 10's partial-answer-beats-500 rule). A less-perfectly-ranked feed beats a spinning loader. Ranking is an enhancement on a feed that must render regardless.

Every failure: named, bounded, answered in advance, and the biggest one (the celebrity write) is answered by the architecture itself, not a runbook.

Security and Abuse: The Part Everyone Skips

A feed is a distribution engine, which makes it an abuse engine unless designed otherwise (defensively, per Part 11): blocking and muting must filter the feed (a blocked account's posts never enter the target's candidate pool, enforced at generation, not hidden at render, where a client bug re-exposes them); privacy-aware fan-out (a private account's post fans out only to approved followers, visibility checked at fan-out and at read — the two-gate rule, a follower removed after the push must still be filtered on read); spam and bot rate limits (post creation and follow actions throttled per account, Parts 10, 11 — a follow-spam bot is a feed-pollution machine); report and takedown flows (a removed post must disappear from already-fanned-out feeds, the invalidation path matters as much as the distribution path — Bonus H's rebuild button, aimed); ranking manipulation resistance (engagement signals are abuse targets — fake likes to game distribution — anomaly detection belongs in the signal pipeline); no sensitive data in the feed's logs and analytics (Part 12 — who-sees-what is sensitive metadata); and amplification awareness (the same fan-out that spreads a good post spreads a harmful one to millions in seconds — rapid takedown is a safety mechanism, not just a feature).

The Observability Plan

Part 12's table, at feed scale:

SLO:           99.9% of feed reads served < 200 ms
Feed reads:    p95/p99 latency, cache hit rate, pull-path merge time
Fan-out:       insertions/sec, worker lag, push-vs-pull split ratio,
               celebrity-threshold crossings
Freshness:     post→feed propagation delay (normal vs celebrity paths)
Ranking:       candidate-set size, heavy-rank latency, degradation events
Counters:      shard distribution, aggregation lag, approximate-vs-true drift
Stores:        per-shard hot-key detection (Bonus G), post-store read load
Business:      feed engagement, scroll depth, time-to-first-post-render
Synthetic:     robot users (normal + celebrity-following) loading feeds per region

The push-vs-pull split ratio is the metric to frame: it tells you whether your celebrity threshold is tuned right. Drifting toward all-push means the threshold's too high (fan-out costs climbing); toward all-pull means it's too low (read costs climbing). The dial has a gauge.

What Interviewers Actually Probe

Feeds are the most-asked hard system design question. "Fan-out on write or read?" — neither alone; hybrid, split at a follower threshold; leading with the trade-off table and then the hybrid is the answer that separates senior from mid. "How do you handle a celebrity with 50M followers?" — pull, not push; the post written once and cached, read by millions; connect it to Part 13's "read-heavy immutable object is the cache's favorite meal" and you've shown the series clicking together. "How is the feed ranked without blowing the latency budget?" — staged ranking: shrink the candidate set, then spend the expensive model on survivors. "5 million likes on one post — the counter?" — sharded/approximate counters, unlocked by the accuracy NFR; name the NFR as the enabler. "A normal user goes viral and crosses the threshold." — migration on next post plus monitoring accounts nearing the line. "How fresh is the feed?" — eventually consistent, seconds for normal, longer for celebrity, and why that's acceptable (fan-out never sat on the read path). "What breaks first at 10×?" — fan-out workers, then feed-cache economics, then ranking cost, in order, each with its metric.

The meta-move: name the read-vs-write trade explicitly. Every feed decision — push/pull, ranking stage, counter design — is that one trade in a new costume. Candidates who say so out loud sound like they understand feeds structurally, not by recipe.

What We Deliberately Didn't Build

The ranking ML itself — model architecture, feature engineering, training pipelines: a whole discipline; we designed the system shape that hosts it, not the model. The full social graph service — "who follows whom" at 100M users is its own storage and traversal problem (graph-database territory, Bonus H); we consumed it as a dependency. Content moderation systems — safety-critical, largely separate; we provided the takedown hooks, not the detection. Stories / ephemeral content — a variation on the pattern (fan-out + expiry), not a new problem. Multi-region feed distribution — genuinely global feeds route users to regional infrastructure; how they reach the nearest edge is DNS/Anycast territory (Bonus D, when it lands, and it's due right about now). Direct messaging — that was Part 15; a feed is not a chat, though products bundle them.

Common Mistakes Engineers Make with This Problem

Picking pure push or pure pull — one breaks on the celebrity's write, the other on the heavy reader's read; the hybrid exists because neither survives alone. Pushing celebrity posts to millions of feeds — the single most expensive mistake in feed design, the meltdown the threshold prevents. A single counter row for viral likes — Bonus G's hot key, unmitigated; sharded/approximate or it melts. Ranking the entire candidate universe at read time — the latency budget, incinerated, stage it. Treating the feed cache as truth — it's derived and rebuildable, the post store is the source of truth (Bonus H). Strict-chronological assumptions in a ranked world — ranking makes feeds non-deterministic, designs assuming stable order break. No celebrity-threshold migration — the account that outgrows push overnight with no path to pull. Exact counters when approximate was allowed — paying for coordination the NFR never required. Blocking enforced at render, not generation — a client bug re-exposes blocked content, enforce in the pipeline. Ignoring takedown propagation — distribution designed, deletion forgotten, the removed post that lives on in fanned-out feeds. Synchronous fan-out on the post path — making the poster wait for N insertions, fan-out is async, always (Part 7). No push-vs-pull ratio monitoring — the threshold dial with no gauge, drifting toward a cost cliff nobody sees.

From My Journey: An Architectural Lesson

The feed is the system everyone assumes they understand. Show a normal-sized example — a few hundred followers, posts appearing in feeds — and it looks like a straightforward join: your feed is the recent posts of the people you follow. Many engineers design exactly that, and it works beautifully in the demo.

Then one number breaks it: a follower count with too many zeros.

The realisation that reorganizes the whole problem: the average case and the extreme case are not the same system. A design tuned for the 200-follower user is destroyed by the 50-million-follower user, and a design built to survive the celebrity is wastefully heavy for everyone normal. The feed isn't one architecture — it's two architectures, applied per account, split at a threshold that itself has to be watched and tuned.

And every part of it turned out to be the same trade in disguise. Push versus pull moves cost between write and read. Staged ranking moves cost between precompute and query. Sharded counters move cost between accuracy and scale. There was never a design that made everything cheap — only a design honest about where each cost was placed, and why.

The lesson that stuck: a feed is not a list of recent posts. It is a negotiation between the cost of writing and the cost of reading — settled differently for every user.

Key Takeaways

• The feed's defining fact is fan-out amplification: one post can mean tens of millions of feed writes, distributed savagely unevenly.
• Fan-out on write makes reads cheap and celebrity posts catastrophic; fan-out on read makes writes cheap and heavy-follower reads expensive — a pure physics trade (Bonus C).
• Real feeds are hybrid: push for normal users, pull for celebrities, split at a tuned follower threshold — each strategy applied where its economics win.
• The celebrity post written once and cached, read by millions, is Part 13's "read-heavy immutable object" insight resurfacing at feed scale.
• Ranking is staged — shrink the candidate set, then spend the expensive model on survivors — to keep personalization inside the latency budget.
• Viral counters use sharded/approximate designs, unlocked by an accuracy NFR, not a technology choice.
• Feeds are derived and rebuildable; the post store is the source of truth (Bonus H); freshness is eventual by design (Bonus I).
• The threshold is a monitored dial, not a constant — the push-vs-pull ratio is its gauge.
• Distribution and deletion are equal: blocking, privacy, and takedown enforce in the pipeline, not at render.

Interview Lens

Feeds are the flagship hard problem, so the structural move matters more than any single fact. Lead with the trade-off, not a choice: "There are two approaches — push and pull — and each breaks at one extreme; here's the table, so real systems go hybrid." That sequence — problem, trade, synthesis — is the difference between reciting an answer and deriving one in front of the interviewer.

Then let the numbers force it: the celebrity's fan-out arithmetic makes the hybrid inevitable, not clever. Volunteer the failure walk — celebrity at peak, cold feed cache, the account crossing the threshold — and name the read-vs-write trade each time it reappears.

The meta-skill: recognizing one trade-off in many costumes. Push/pull, ranking stages, counter sharding — all the same negotiation between write cost and read cost. The candidate who names the pattern, not just the parts, is the one who's understood feeds.

Real-World Engineering Lens

Find the fan-out in your own system. Anywhere one write triggers many — notifications, feeds, activity logs — ask: push or pull, and what happens at the extreme? The celebrity problem hides in more systems than social apps.

Put the amplification factor on a dashboard. Writes-per-action, per account tier. The number that's fine on average and lethal at the tail is the one to watch (Bonus G's whole lesson).

Tune thresholds with data, not guesses. The push/pull cutover is a dial — instrument the ratio, move it deliberately, never hardcode it once and forget.

Let the NFR unlock the cheap design. Before building exact anything at scale, ask what accuracy the product actually needs. "Approximate is fine" is the most cost-saving sentence in system design.

Design deletion alongside distribution. Every fan-out path needs a matching invalidation path — build them together, or explain the orphaned content later.

What's Next

The feed distributes text and references at massive scale. But some of those references point at the heaviest payload in computing — video — and delivering that at scale is a different beast: encoding, adaptive bitrates, and edges around the world.

Getting bytes to a global audience means getting users to the nearest edge, which is a DNS and Anycast problem the series has deferred long enough.

Bonus D — DNS Internals, Anycast, and Global Traffic Routing. The last unwritten bonus, finally due — how a user in Mumbai and a user in Toronto hit the same domain and reach entirely different machines.

Then the main series continues with Blog 17 — Media Streaming and Recommendations: CDN, ABR, and Personalisation.

Question for Readers

Find one place in your system where a single action triggers many downstream writes — a fan-out, however small.

Is it push or pull? And what happens the day one source of those writes gets 100× bigger than the rest?

That account, that key, that tenant — the extreme case you haven't met yet — is already deciding your next redesign.