Storage Systems

"Just put it in S3." Six words. A dozen hidden decisions.

Part 7 ended with an observation: messages move references, something else must hold the bytes. Bonus I sharpened the question: replicas, indexes, projections, caches — all of them are copies of bytes, living somewhere, being shipped somewhere else.

So today we finally answer it: where do the bytes actually live? The video a user uploads. The invoice PDF. The profile photo. The backups. The event log Part 7's streams quietly retain.

Most engineers answer with a shrug and a bucket name. This article is about everything that shrug is hiding.

What "Storage" Means in System Design

Back in Blog 5, we split data into structured, semi-structured, and unstructured, and noted that unstructured data (videos, images, documents) is often much larger than everything else combined. Blog 5 and Part 6 dealt with the structured world: rows, queries, caches. This article is where the unstructured world lives.

And here's the mindset shift: storage is not "where files are kept." Storage is a design decision about how bytes are written, read, named, replicated, secured, moved, archived, and recovered. Different bytes deserve different answers. Choosing the answer is the job.

The Simple Mental Model: Warehouse, Locker, and Raw Disk

There are three fundamental shapes of storage. Keep these three pictures.

Object storage is a massive warehouse. Every item gets a unique label (a key). You hand the label to the warehouse, the warehouse hands you the item.

"user-uploads/2026/invoice-8841.pdf"  →  [the bytes]

No wandering the aisles, no folder-by-folder traversal — access is by key. Those slashes in the key look like folders; they aren't. Object storage has a flat namespace, and the slashes are a costume that makes humans comfortable.

File storage is a shared office cabinet. Folders inside folders, paths, permissions on directories:

/shared/projects/2026/reports/q2.docx

Multiple people, and applications, open the same cabinet and navigate the same hierarchy. Access is by path.

Block storage is raw building material. A slab of raw blocks attached to one machine. The storage itself has no idea what a "file" is — the operating system shapes it into something usable, the way a builder shapes raw material into rooms. Access is by block, and almost always by exactly one server.

And "distributed" is a property, not a fourth shape. Distributed storage means the data is spread across many machines, so no single disk, server, or rack is a point of failure, and capacity can grow past what one machine holds. Modern object storage is almost always distributed — that's precisely where replication and erasure coding, coming shortly, do their work. (Designing a distributed storage system from the inside is Part 20's job. Today we're choosing between them from the outside.)

Database vs Object Storage: Why Not Put Everything in the Database?

The most common storage question in real teams: a user uploads a 20 MB training video, or an invoice PDF, or a product image. "We have a database. Why not store it there?"

Because databases and blobs want different things. A relational database is optimized for many small structured records — fast lookups, transactions, indexes, careful page-by-page memory management (Part 6's buffer pool). Feed it 20 MB blobs and everything it's good at gets worse: backups and replication bloat, since every replica (Bonus I) now ships gigabytes of video around. Memory suffers, as blobs elbow hot rows out of the buffer pool. Connections get hostage-taken — a database connection streaming video is a connection not answering queries. You can't CDN a table row, so delivery options shrink. And cost: database storage is expensive real estate; object storage is warehouse space.

The Pattern That Actually Works

Split the responsibilities: object storage holds the bytes, the database holds the metadata. The metadata row looks like this:

object_key      "user-uploads/2026/invoice-8841.pdf"
file_name       "March-Invoice.pdf"
content_type    "application/pdf"
size_bytes      2048576
owner_id        user_5521
access_policy   "private"
checksum        "sha256:9f8a…"
upload_status   "complete"
created_at      2026-07-07T10:15:00Z

The database remains what Blog 5 said it must be: the source of truth — about what exists, who owns it, and who may touch it. The object store is the payload.

The Honest Caveat

This is a default, not a law. Tiny files (a few KB — icons, thumbnails) or blobs that must live and die inside a strict transaction can justify database storage; some databases handle small blobs perfectly well. But for large binary content, the split is the industry default for good reasons — and the rest of this article assumes it.

Object Storage: The Default Home for Blobs

Let's give the warehouse a proper tour.

The vocabulary. Object — the bytes plus their metadata, as one unit. Bucket / container — the top-level namespace objects live in. Key — the object's unique name within the bucket, the label on the warehouse item. Metadata — content type, custom tags, checksums, riding along with the object.

The behaviors that matter.

You don't edit objects — you replace them. Object storage favors whole-object writes; updating means uploading a new version, not editing bytes in place. With versioning enabled, old versions survive, which turns "we overwrote the contract PDF" from a disaster into a rollback. Sound familiar? It's Part 6's versioned-key trick (app.4f9a2c.js) — change the content, change the identity.

Consistency has history. Some object stores historically served eventually consistent reads or listings, where an upload could take a moment to appear in list results. Major platforms have since strengthened this considerably, but the honest engineering habit remains: check your platform's current consistency model before your design depends on it.

Durability is the headline promise. Major providers advertise extraordinary durability figures — commonly cited as "eleven nines," 99.999999999% — by spreading copies across machines and facilities. What that promise does and doesn't mean gets its own section shortly.

Signed URLs: Temporary Permission, in a Link

Private objects create a problem: the bucket isn't public, but a user needs this one file, right now. The answer is a signed URL:

App verifies: user may access invoice-8841
   ↓
App generates a URL containing:
   the object key + an expiry time + a cryptographic signature
   ↓
User's browser downloads DIRECTLY from object storage
   ↓
After expiry → the same URL is useless

Conceptually: a visitor pass. Anyone holding it gets in, until it expires. Two design consequences follow: the bytes never flow through your application servers, since object storage does the heavy lifting (uploads can work the same way, with signed upload URLs); and the expiry is the security decision — a 7-day pass to a salary PDF is a leak with a countdown timer.

CDN + Object Storage: The Delivery Duo

Part 4 introduced the CDN; Part 6 made it cache layer #2. Here's its natural partner:

Object storage  =  the origin (one authoritative copy)
CDN             =  the delivery network (copies near users)

Public, cacheable content — product images, video segments, static assets — is served from CDN edges, with object storage consulted only on cache misses. Versioned keys keep invalidation trivial, exactly as Part 6 taught.

What lives here. Images, videos, PDFs, backups, logs, ML datasets, static website assets, Part 7's retained event archives. If it's a blob, this is its default home.

Block Storage: The Disk Your Server Thinks It Owns

Block storage is the least visible of the three, and the one everything else secretly stands on. A block volume is a virtual disk attached to a server or VM. The server formats it, mounts it, and treats it as its own private drive. The operating system does file management; the block layer just reads and writes numbered blocks, fast.

What it's for. Database volumes — the PostgreSQL from Blog 5 stores its tables somewhere, and that somewhere is a block volume, chosen for low-latency random reads and writes. Boot disks and VM storage. Anything an operating system needs to treat as a local disk.

What it's not for. Sharing. A block volume typically belongs to one server at a time — it's a disk, not a shared drive. Ten application servers cannot casually mount one volume and coordinate through it.

That's the tell: block storage is infrastructure-layer storage. Your platform team thinks about it constantly. Your application code should almost never talk to it directly.

File Storage: Shared Folders for Applications

File storage is the familiar one: directories, paths, and POSIX-style file access — but served over the network, so many servers can mount the same folder tree.

Where it earns its place. Legacy applications that expect a filesystem and won't be rewritten. Shared document workflows. Media-processing pipelines where one stage writes /renders/scene-42/ and the next stage reads it. Anything that genuinely needs "many machines, one folder tree."

Where it hurts at scale. The cabinet gets crowded: locking (two writers, one file, who wins?), concurrent writes (POSIX semantics get expensive when enforced across a network), metadata bottlenecks (listing a directory with ten million files can hurt more than reading them), permission management (nested directory ACLs grow into archaeology), and network latency (every file operation is now a network operation wearing a filesystem costume).

A useful rule: file storage is chosen for compatibility, object storage for scale. When an app needs paths, use file storage. When you're designing fresh, reach for objects.

The Three, Side by Side

Durability vs Availability: Not the Same Thing

Two words engineers use interchangeably. They must not.

Durability: the data is not lost. Availability: the data can be accessed right now.

Read that second row twice — it's Bonus I's asynchronous window with teeth, and Part 7's write-behind warning made literal: an acknowledgment is only as strong as what has actually been replicated at the moment of the ack.

Durable-but-unavailable is an incident. Available-but-not-durable is a tragedy you discover later.

How Storage Earns Durability: Replication vs Erasure Coding

Two strategies, one goal — surviving dead disks and dead machines.

Replication: keep full copies.

Object → Copy A (machine 1) + Copy B (machine 2) + Copy C (machine 3)

Simple. Fast reads. Costs 3× the storage.

Erasure coding: split the object into fragments plus computed parity fragments; any sufficient subset rebuilds the whole.

Object → 10 data fragments + 4 parity fragments, spread across 14 machines
         Any 10 of the 14 can reconstruct it

Survives 4 simultaneous losses at roughly 1.4× storage instead of 3×. The price: more CPU, and slower reconstruction when things fail.

The conceptual rule of thumb: replication for hot, small, latency-sensitive data; erasure coding for big, cooler data — which is exactly why object stores love it.

Not All Bytes Deserve the Same Home: Tiers, Lifecycles, and Cost

Here's the question teams forget to ask: how often will these bytes be read?

Last week's user uploads: hot. Last quarter's logs: warm. The 2019 compliance records: cold.

Lifecycle policies: the automation. Nobody manually moves a billion objects. A lifecycle policy is a standing rule:

Logs:  hot for 30 days → warm for 90 → archive → delete at 2 years

Set it when the bucket is created, not when the bill arrives.

Backup vs archive: not synonyms. Backup is for recovery — recent, restored fast, tested regularly, answering "we deleted production data at 2 PM; get it back." Archive is for history and compliance — old, cheap, slow, rarely touched, answering "the auditor wants 2021." Confusing them has a failure mode in each direction: backups rotting in a tier too slow to restore from in an emergency, or someone "cleaning up old backups" that were legally mandated archives.

The four costs. Storage bills have four line items, and teams reliably plan for only the first: storage (GB per month — the one everyone estimates), requests (per operation — a million tiny files cost real money just to touch), retrieval (the cold-tier toll booth), and egress/bandwidth (bytes leaving the platform — the classic bill shock, whether from serving popular video straight from origin, region-to-region copies, or "quick" data migrations). A CDN with a healthy hit rate (Part 6) shields the origin from repeated fetches — one of the quiet financial arguments for the caching stack.

The Storage Design Pattern Most Engineers Use

Assemble everything above and you get the architecture that powers most upload features on the internet:

User uploads file
   ↓
Application validates metadata      (type, size, permissions)
   ↓
Object storage stores actual bytes  (durability, scale)
   ↓
Database stores metadata + key      (source of truth)
   ↓
CDN serves public/cacheable content (speed, egress relief)
   ↓
Signed URL serves private content   (temporary permission)

Each part does the one job it's best at: the application is the bouncer and bookkeeper (authenticates, validates, issues keys and signed URLs, owns workflow state — it should not be the pipe the bytes flow through), object storage is the warehouse (holds and streams the bytes), the database is the registry (what exists, who owns it, who may see it, whether the upload actually completed), the CDN is the delivery fleet for anything public and cacheable, and signed URLs are the visitor passes for everything private.

Two production refinements worth knowing. Direct-to-storage uploads: mature systems have the client upload straight to object storage using a signed upload URL — the application approves the upload and records it, but never carries the 2 GB video on its own back. The pending → complete handshake: create the metadata row as pending, let the upload happen, then verify (size, checksum) and flip to complete. Reads only trust complete rows; a cleanup job sweeps stale pending ones. This one habit prevents half the failure modes in the next section.

What Breaks at Scale

The pattern is sound. Scale still finds the seams.

Millions of small files. Request costs eclipse storage costs; listings crawl. Sometimes the fix is batching small things into larger objects.

Very large files. A 5 GB upload over one connection will fail at 97%. Large uploads are done in parts (multipart), which introduces its own orphan: the incomplete upload nobody finished or aborted, quietly billable until a policy cleans it up.

Upload retries and duplicates. A retried upload is Part 7's duplicate delivery wearing a file extension. Idempotency by content — same checksum, same object, not a second copy.

Broken references, both directions. A DB row without an object is the 404 profile photo. An object without a DB row is an invisible orphan you pay for monthly, forever. You need a sweeper that reconciles both ways.

Metadata database bottleneck. Ironically, the database holding tiny metadata rows often strains before the object store holding petabytes — every upload, listing, and permission check hits it. Blog 5 and Part 6 apply: index it, cache it, and eventually shard it, carefully.

Storage hotspots from key design. Keys decide where data lands. Name every object 2026-07-07-<n> and today's writes crowd one range of the keyspace while the rest idles. Sequential keys, timestamps-first, celebrity prefixes — same disease. Hold that thought. It's about to get its own article.

CDN cache invalidation. Part 6's second-hard-thing, now for files — versioned keys remain the honest answer.

Access-control mistakes. The accidentally-public bucket is a genre of security incident at this point. So is the signed URL with a month-long expiry, forwarded in an email chain.

Lifecycle policy mistakes. Wrong prefix in an expiry rule and production assets quietly delete themselves at midnight. Lifecycle rules deserve code review, not console clicks.

Uncontrolled cost growth. No lifecycle plus accumulating orphans plus unmonitored egress equals the bill that summons a VP.

Common Mistakes Engineers Make with Storage

Defaulting large blobs into relational tables — the database is for the registry, not the warehouse. Storing files without metadata discipline — bytes with no checksum, owner, or status aren't stored, they're abandoned in place. Treating object storage like a file system — there are no folders, renames aren't moves, and directory-style listing is a query, not a stroll; the slashes are a costume. Ignoring signed-URL expiry — the expiry is the permission, and long-lived links to private data are leaks on a timer. Never cleaning orphans — every failed upload and deleted-row-without-deleted-object accrues cost and risk monthly. Skipping size/checksum verification — an unverified upload is a rumor, not a file. Budgeting only for storage cost — requests, retrieval, and egress are three-quarters of the bill's surprises. One storage class for everything — paying hot-tier prices for 2019's logs is a subscription to waste. Conflating backup and archive — one is for emergencies, one is for auditors; design them separately. Ignoring key naming — keys decide placement, placement decides hotspots, the next article's whole subject.

Applying C.R.E.D to Storage

Clarify — What kind of data is this? Blob or record? Public or private? How often will it actually be read, and by whom?

Requirements — Durability level, latency, access control, retention, compliance. Per data type — the same discipline as Part 6's TTLs and Bonus I's staleness budgets.

Estimate — Object count and average size (request costs live here), growth per month, read/write ratio, egress volume. The bill is an estimation exercise you do now or a surprise you get later.

Design — Storage shape (object/block/file), tier and lifecycle, metadata schema with status and checksum, delivery path (CDN, signed URLs), and the sweeper that keeps references honest.

Fill those blanks and you've stopped "saving files." You're designing storage. A good architect does not ask only, "Where should we store this file?" A good architect asks, "What kind of data is this, how will it be accessed, how durable must it be, how fast must it be, and what will it cost at scale?"

From My Journey: An Architectural Lesson

In many applications, file handling starts as a small requirement. Upload a document. Attach an image. Save a report. One ticket among fifty.

The first instinct is almost always the same: treat the file like any other piece of data and push it through the same backend and database path everything else uses. It works in the demo. It even works in the first month of production.

Then the files do what files do: they multiply, and they get heavier.

The realisation that follows is the one this article is built on: files behave differently from rows. They're larger, slower to move, costlier to serve. They need their own access rules, their own metadata, their own delivery path, their own lifecycle. A row is a fact; a file is a shipment.

Splitting the responsibilities — bytes in object storage, metadata in the database — cleaned up the design the way it always does. But it didn't remove the design work; it changed its shape. Now there were references that could break, orphans that could accumulate, signed URLs that could outlive their welcome, cleanup workflows nobody had assigned an owner.

The lesson that stuck: storage design is not just about saving bytes. It is about deciding who owns the bytes, who owns the metadata, who can access them, and how long they must live.

A file is not just data. It is data with weight, location, access rules, and a lifecycle.

Key Takeaways

• Storage is a design decision, not a destination — access pattern, durability, latency, and cost pick the shape.
• Object = warehouse by key; block = raw disk for one machine; file = shared folders by path; "distributed" is the property that makes them survive machine failure.
• The default pattern: bytes in object storage, metadata (with key, checksum, and status) in the database as the source of truth.
• Signed URLs are temporary permission in a link — the expiry is the security decision.
• Durability (not lost) and availability (reachable now) are different promises; an ack is only as strong as what's actually been replicated.
• Replication buys simplicity at 3×; erasure coding buys efficiency with more moving parts.
• Hot, warm, and cold tiers plus lifecycle policies are how storage stays affordable; backup and archive are different jobs.
• The bill has four line items — storage, requests, retrieval, egress — and egress is the classic surprise.
• Keys decide placement, and placement decides hotspots — which is exactly where we go next.

Interview Lens

Storage rarely gets its own interview question. It hides inside "design a photo-sharing app," "design YouTube," "design a document service" — and the upload pipeline is where candidates quietly separate.

The weak answer: "Files go to S3." The strong answer walks the pattern in one breath: "Client uploads directly to object storage via a signed upload URL; the metadata row is created as pending and flipped to complete after checksum verification; public assets are served through the CDN with versioned keys; private files get short-expiry signed URLs; lifecycle rules move uploads to colder tiers after ninety days."

Expect the probes: "Why not store images in the database?" — backups bloat, buffer-pool pollution, hostage connections, no CDN path, then the caveat that tiny files or strict transactional coupling can flip the answer. "How do you handle a 5 GB upload?" — multipart, resumable, and a cleanup policy for abandoned parts. "How are private files served?" — signed URLs, and name the expiry trade-off unprompted. "What's your durability story?" — replication or erasure coding across failure domains, and the durability-vs-availability distinction. "What does this cost at 100 million files?" — the four line items; mentioning request costs and egress is an instant differentiator, because almost nobody does.

Real-World Engineering Lens

Write the metadata schema before the first upload ships — key, size, checksum, owner, status, created-at. Retrofitting discipline onto a bucket of mystery objects is archaeology.

Build the sweeper early. One scheduled job, two questions: which rows point at missing objects, and which objects have no rows? Both counts should be on a dashboard, trending toward zero.

Default signed URLs to short expiries — minutes, not days. Lengthen deliberately, per case, with a reason written down.

Create lifecycle rules the day the bucket is born, and put them through code review — a mistyped prefix in an expiry rule is a production deletion incident on a schedule.

Do the restore drill. A backup that has never been restored is a hope, not a backup. Time the restore; that number is your real recovery capability.

Put egress on the cost dashboard, next to the CDN hit rate — the two numbers explain each other.

What's Next

One thread from this article deserves immediate attention. We said it twice: keys decide where data lives. Name your objects — or your database shards — carelessly, and all the traffic crowds one place while the rest of the system idles. Blog 5 introduced sharding and left a warning about choosing the shard key. This article added the object-storage version of the same disease. Time to treat it properly.

Bonus G — Shard Key Design and Hotspot Prevention. Why "just shard by user ID" sometimes works, sometimes melts a partition, and how to tell in advance.

After that, the main series continues with Blog 9 — Scaling Compute: Microservices, Containers, and Serverless. The bytes have a home. The messages have a broker. Next: the machines that actually run your code, and how they multiply.

Question for Readers

Pick one file your system stores today — an upload, an export, a report. Can you answer all four?

Where do its bytes live? Where is its metadata, and does it include a checksum and status? Who can access it, and until when? When will it be deleted, and by what rule?

Any "not sure" in that list is your storage design backlog, in priority order.