Security and Authentication

Every credential in your system is a promise about trust. Most systems have no idea how many promises they've made.

Part 10 ended at the vault door. The gateway checks every visitor's pass — but we never explained where passes come from, what they mean, how long they last, or what happens when one leaks. Time to open the vault.

And here's the reframe that carries this whole article: security is not a login feature bolted on at the end. Security is the architecture of trust — who is calling, what they may do, how that trust is issued, proven, rotated, revoked, and logged, and where the master keys live.

Every system you've built has this architecture. The only question is whether anyone designed it.

The Simple Mental Model: Building Entry, ID Card, and Room Access

Picture a corporate office building.

Authentication is showing your ID at the entrance — the guard confirms you are who you claim to be. Authorization is which rooms you may enter — being inside the building does not open the server room. A token is a temporary access badge, issued after your ID check, carried on you, checked at doors, and it expires. A refresh token is the renewal counter, a way to get a fresh badge without repeating the full entrance ceremony. SSO is the central corporate front desk trusted by many buildings — one ID check, badges that work across the campus. Secrets are the master keys that systems themselves carry, and master keys demand careful storage and regular rotation.

Five principles fall straight out of the building: identity proves who you are, access decides what you can do (different questions, different systems); trust must expire, since a badge valid forever is a breach waiting for its moment; trust must be revocable, since people leave and badges must die; not every authenticated person opens every room, since inside doesn't mean allowed; and the one teams forget — systems carry keys too, and machines' keys need more care than users' passwords, because machines never notice theirs are missing.

Authentication vs Authorization: The Confusion That Breaks Systems

The two words engineers swap carelessly, and the swap has a body count.

Authentication answers: "Who are you?" Authorization answers: "What are you allowed to do?"

Watch them separate in practice: a user is logged in but cannot open the admin panel — authenticated, not authorized. A service is authenticated but has no permission to call the payment API — known, not trusted with money. An employee belongs to Tenant A and must see nothing of Tenant B — real identity, bounded access. An API key is valid but has exceeded its quota, or was never granted that scope — recognized, still refused.

The one-liner worth keeping: authentication without authorization is just knowing the wrong person very confidently.

One more widening before we go deeper: identities are not just humans. Users, admins, partner integrations, background jobs, and services themselves all carry identity, and every trust decision in this article applies to all of them. Hold that; it returns in the service-to-service section.

Passwords and MFA: The Oldest Doors

Before badges, the entrance itself — briefly, because two rules here are non-negotiable.

Never store plain-text passwords. Not once, not temporarily, not in logs. The database should not know the password, only proof of it. That proof comes from hashing, not encryption, and the difference matters: encryption is a locked box that can be opened with a key; a hash is a one-way fingerprint that cannot be reversed. Passwords get salted, slow, purpose-built hashing — salted so identical passwords produce different fingerprints, deliberately slow so bulk guessing is expensive by design.

Multi-factor authentication strengthens the entrance itself: something you know (password) plus something you have (a device, a code). One stolen factor is no longer enough — which is the entire point.

Now — what does the entrance hand you once you're through?

Sessions, Cookies, Tokens, and JWTs

Two great traditions of "proving you already checked in."

The Session: The Guard Remembers You

Classic server-side sessions: on login, the server creates a session record and hands the browser a session ID in a cookie. Every request carries the cookie; the server looks up the record.

The state lives on the server — which, after Part 9, should ring a bell: a fleet of stateless instances means the session store must be external and shared (Part 6's distributed cache is its classic home). Do that, and sessions scale fine, with one superpower we'll miss later: delete the record, and the user is out. Instantly.

The Token: The Badge Speaks for Itself

Bearer-token auth flips it: the server issues a self-contained badge; whoever bears it is treated as its subject. No lookup — the badge carries its own claims and proof.

The dominant format is the JWT:

eyJhbGciOi…  .  eyJzdWIiOi…  .  SflKxwRJSM…
   HEADER          PAYLOAD          SIGNATURE
(how it's signed) (the claims)  (proof of the issuer)

Inside the payload — the claims:

sub:   user_5521          (subject — who this is about)
iss:   auth.example.com   (issuer — which front desk made it)
aud:   orders-api         (audience — which building it's for)
exp:   1720340000         (expiry — when it dies)
iat:   1720336400         (issued at)
scope: orders:read        (what it permits)

Three facts about JWTs that prevent most JWT incidents. It is signed, not encrypted — the payload is merely encoded, and anyone holding the token can read every claim; the signature proves who issued it and that nothing was altered, but it hides nothing. Never put data in a JWT you wouldn't show its bearer. Validation is the entire security — a JWT is only as trustworthy as the checks applied to it: verify the signature, the expiry, and the audience, every time, at every service that accepts it. A payload read without validating the signature is a costume, not a credential. And bearer means bearer — whoever holds it, is it, which makes storage and transport part of the design: always over TLS, kept out of URLs and logs, and short-lived, so a leaked badge is a small window, not a skeleton key.

Side by Side

That revocation row is the next section's whole subject.

Access Tokens, Refresh Tokens, and the Revocation Problem

Short-lived badges are safer, but nobody wants to re-enter the building every fifteen minutes. The industry's answer is a pair: the access token, the short-lived badge (minutes to an hour) presented to APIs on every request; and the refresh token, the longer-lived renewal credential, used only at the front desk to obtain fresh access tokens.

The division of risk is the design: the access token travels everywhere and expires fast; the refresh token travels rarely, and therefore must be protected far more carefully, stored securely, and ideally rotated on every use, so a stolen one betrays itself the moment two parties try to renew with it.

Expiry limits damage. Rotation shrinks windows. Those two habits do more for token security than any exotic mechanism.

Revocation Is the Price of Statelessness

A session dies when you delete its record. A JWT has no record to delete — every service that trusts the signature will keep accepting it until it expires, no matter what happened in between.

Which means logout does not automatically mean what users think it means. Deleting the token from one device removes that copy. The credential itself may live on until exp.

The practical toolkit, conceptually: keep access tokens short (the window is the exposure), track and revoke refresh tokens server-side (killing the renewal path ends the account's future), and for genuinely critical events — offboarding, compromise — maintain a revocation check for the brief remaining lifetime of outstanding tokens.

And notice the shape of this problem. It's Bonus I's permissions lag, wearing a badge: the decision to revoke happened at the source of truth; copies of trust are still honored elsewhere until they catch up. Same disease, a security hole with a timestamp. Design the window's length on purpose.

One more familiar face: Part 8's signed URLs — temporary, scoped, expiring, self-proving. You've already been designing good tokens. You just called them visitor passes.

OAuth2, OpenID Connect, and SSO in Plain English

Three acronyms, one plain-English tour, deliberately not a standards tutorial.

OAuth2: Acting on Your Behalf, Without Your Password

The problem it solves: an app needs to access something of yours at another service — your files, your calendar — and the catastrophic pre-OAuth answer was "give the app your password."

OAuth2's answer: delegated authorization.

You → App: "fetch my files from CloudDrive"
App → sends you to CloudDrive's AUTHORIZATION SERVER
You → prove yourself to CloudDrive (the app never sees this) and consent
Authorization server → issues the app a scoped ACCESS TOKEN
App → presents the token to the RESOURCE SERVER (the files API)

The app holds a badge that says "may read this user's files," never the keys to your whole identity. The scope on the token is the leash.

OpenID Connect: The "Who" Layer

Strictly, OAuth2 answers what an app may do, not who the user is. OpenID Connect (OIDC) adds the identity layer on top: alongside the access token comes an ID token, a signed statement of who authenticated and when.

The two-line separation worth memorizing: OAuth2 delegates access. OIDC asserts identity. "Log in with Google" is OIDC riding on OAuth2's rails.

SSO: One Front Desk, Many Buildings

Enterprises run dozens of applications. Without central identity, that's dozens of passwords, dozens of account databases, and worst of all, dozens of places to forget when someone leaves.

Single Sign-On centralizes it: the identity provider (IdP) is the corporate front desk, the one place identity is proven; the service providers are the applications, which trust the desk's word instead of running their own ID checks.

One login, badges across the campus. But the architectural prize is the user lifecycle: onboarding grants access everywhere from one place, and offboarding is one switch, not an archaeology dig through fifteen per-app account systems. (And per Bonus I: check how fast that switch propagates. Revocation lag on a departed employee's access is exactly the dangerous kind.)

Centralized identity is also a dependency — the front desk goes down, every building feels it. That's a trade you make with eyes open, not by accident.

API Keys: Useful, But Not User Identity

The humblest credential deserves an honest job description.

What API keys are good at: identifying a service or integration — the partner calling your API, the analytics job, the machine-to-machine caller in a controlled setting. They plug directly into Part 10's machinery: rate limits, quotas, and per-client usage tracking all key off them. Simple to issue, simple to check.

What they're not: user identity. A key says which integration is calling — it carries no user consent, no fine-grained "on behalf of," no session, no MFA.

And their occupational hazards: they tend to be long-lived (leaks stay valuable), they end up in code and config files (we'll get to secrets), and without discipline they become immortal. The discipline: every key gets an owner, a scope, a rotation schedule, and monitoring — a key nobody owns is Part 7's unowned queue, wearing credentials.

Authorization Models: RBAC, ABAC, Scopes, and Tenants

Authentication told us who. Now the harder design space: what may they do?

RBAC — role-based access control. Users get roles; roles get permissions. Admin, editor, viewer. Simple, legible, everywhere — and usually the right starting point. Its ceiling: real rules eventually outgrow role labels ("editors, but only for their own region's documents…").

ABAC — attribute-based access control. Decisions from attributes: department, region, resource ownership, data sensitivity, time of day, tenant. Enormously flexible, and correspondingly harder to reason about, test, and audit. Adopt when the rules demand it, not before.

Scopes and permissions. The API-world vocabulary — orders:read, payments:write — riding inside tokens (that scope claim from earlier). Scopes let a credential carry less power than its owner has: the leash, formalized.

Tenant isolation — the one that ends companies when it fails. In multi-tenant systems, Tenant A must never see Tenant B's data. Two rules, absolute: the tenant boundary is enforced on every query, in the backend, consistently, not as an afterthought filter; and never trust a client-supplied tenant ID — the tenant comes from the verified identity, the validated token, not from whatever the request claims. A request is testimony; the token is evidence.

Service-to-Service Trust

Part 9 split the system into services. Part 10 gave them internal APIs. Which quietly created a new population of callers: the services themselves.

The lazy assumption — "it's on the internal network, it's fine" — is the one this section exists to kill. Internal networks get breached, misconfigured, and shared; an internal caller is not a trusted caller until it proves who it is. Location is not identity.

The toolkit, conceptually: service identity (every service is an actor with a name, like any user), signed tokens between services (the same badge machinery, issued to machines), mTLS (mutual TLS — both sides of the connection prove identity with certificates, not just "am I talking to the real payment service" but "is the caller really the order service"), service accounts with least privilege (the order service can call payments, it cannot read the HR database — over-permissive service accounts are among the most common quiet disasters in microservice estates), short-lived credentials (machines renew badges beautifully, let them — long-lived machine credentials are leaked machine credentials, eventually), and audit logs (services acting on services still leaves a trail, or should).

Machines outnumber your users in most architectures. Their trust design deserves at least equal care.

Secrets Management: Where the Master Keys Live

Every connection in this series has a secret behind it: database passwords, API keys, token-signing keys, private keys, OAuth client secrets, encryption keys, webhook secrets. The master keys of the building.

The rules, in escalating order of organizational maturity. Never in code — a secret committed to a repository is published, to every clone, every fork, every laptop, forever (history remembers even after you delete it). Never in logs — Part 12 will make you log everything, but secrets are the exception that proves the rule; a token in a log file is a credential with a distribution list. Environment variables are a start, not a strategy — they keep secrets out of code, good, but at scale they answer none of the hard questions: who can read them, when were they last rotated, who accessed them yesterday, how do you revoke one everywhere, now?

A secrets vault answers those questions. Conceptually: one access-controlled, audited home for secrets — services fetch what they need at runtime (Part 9's orchestration listed "config and secrets" delivery as a core job; this is what it's delivering), every access is logged, rotation is routine instead of heroic, and emergency revocation is a switch, not a scavenger hunt.

Round it out with least privilege (each service sees only its own secrets), separation by environment (staging keys open nothing in production), and a rotation calendar that treats key age as a liability metric.

Encryption: Important, But Not a Permission System

The most misunderstood word in the chapter — so let's place it precisely.

Encryption in transit protects data moving over the network — TLS, the padlock on every connection this series has drawn. Non-negotiable, everywhere, including internal traffic (see: location is not identity).

Encryption at rest protects data sitting in storage — databases, object storage, backups, logs — so that exposed disks or leaked snapshots yield ciphertext, not customer data. (Part 8 deferred exactly this topic here: the warehouse's contents, locked even inside the warehouse.)

Key management is where both live or die: keys must be protected, rotated, and access-controlled like the crown jewels they are, because the failure modes are symmetric and brutal. Lose the keys, lose the data. Leak the keys, leak the data.

And the point this section exists for: encryption protects data from exposure. Authorization decides who should access it in the first place. Encrypted data served to the wrong authenticated user is a breach with excellent cryptography. Encryption is armor, not judgment — the judgment is everything above this section.

Security at the API Gateway

Back to Part 10's front desk, now that we know what the passes mean.

What the gateway does well: validating tokens once at the door (signature, expiry, audience); forwarding a trusted identity context inward; enforcing coarse-grained access ("may this caller reach this API family at all"); rate limits, quotas, and request size limits — the abuse-protection layer Part 10 built, now understood as security controls; IP allow/deny rules where they genuinely fit; and stamping correlation IDs and logs on every request.

What the gateway must not become: the only security brain. Two rules keep the architecture honest. Business authorization lives in the services — "may this user refund this order" depends on domain state the gateway cannot and should not know; Part 10's rule again: policy at the door, decisions in the rooms. And forwarded identity is only as trustworthy as the path it traveled — a service that accepts an identity header must know that header can only have come from the gateway, a clear trust boundary, network-enforced. A service reachable by other paths that still honors the header has outsourced its security to hope.

Audit: Security's Memory

One more gateway gift, and a bridge to what's next: security events deserve their own log. Logins and failures, permission denials, token issuance, secret access, admin actions, offboarding — stamped with Part 7's correlation IDs.

Because the uncomfortable truth of incidents: you cannot investigate what you never recorded. Part 12 turns this instinct into a discipline.

Common Mistakes Engineers Make with Security and Authentication

Confusing authentication with authorization — the confident wrong person, again. Trusting a JWT's payload without validating its signature — reading the costume, skipping the ID check. Assuming JWT means encrypted — it's readable by every bearer, signed, not sealed. Long-lived access tokens — a big window is a standing invitation. Storing tokens carelessly — in URLs, in logs, in places any script can read. No refresh-token rotation — the one credential worth stealing, left evergreen. Logout that doesn't match token reality — users think "out," the badge says "valid until exp"; design the gap, don't discover it. API keys with no owner or expiry — immortal, orphaned, and eventually leaked. Secrets in code, config files, screenshots, or logs — published, with commit history as the archive. Over-permissive service accounts — machines with skeleton keys they never needed. Weak tenant isolation — the client-supplied tenant ID, trusted; the mistake that ends companies. No audit trail for sensitive actions — an investigation with no evidence. All authorization in the gateway — one brain at the door, empty rooms behind it. No key or secret rotation — credential age as an unmonitored liability. Untested offboarding — the leaver whose badges keep working, SSO's whole promise, unverified. No rate limits or abuse protection — Part 10's guardrails, treated as optional.

Applying C.R.E.D to Security

Clarify — Who are the actors: users, admins, partners, services, jobs, tenants? What must each be able to do, and explicitly not do?

Requirements — Token lifetimes, revocation expectations (what must logout actually mean?), tenant isolation guarantees, audit obligations, secret rotation policy.

Estimate — How many credentials exist, and their ages; the blast radius of each leaked secret; how long revocation takes to propagate (Bonus I's window, measured).

Design — The identity flow per actor, the token pair and lifetimes, the authorization model per boundary, the service-trust mechanism, the vault, the audit trail, and the owner of each.

A good architect does not ask only, "How do we add login?" A good architect asks, "Who are the actors, what are they allowed to access, how is trust issued, validated, rotated, revoked, logged, and protected across the system?"

From My Journey: An Architectural Lesson

In many systems, authentication starts as a login feature. A user enters credentials, receives a token, and starts using the application. One sprint, one checkbox, done.

Then the system grows, and the identities multiply. Users, then admins. Then partners with API keys. Then services calling services. Then background jobs, automation clients, and tenants whose data must never touch. What began as a login feature has quietly become a trust architecture, usually without anyone deciding to design it that way.

The realisation that reframes it: issuing a token was never the hard part. The hard part is everything the token implies — what it means, how long it should live, what it can reach, how it dies before its time, which services may believe it, and where the secrets that mint it are kept.

And every answer is a trade with teeth. Stateless tokens scale beautifully and revoke painfully. Centralized identity gives control and creates a dependency everyone shares. Fine-grained authorization adds safety and complexity in equal measure. Secrets connect everything, and become liabilities the moment nobody manages them.

The lesson that stuck: security design is trust design. Every token, key, role, and secret is a trust decision with a lifecycle.

Authentication opens the door. Authorization decides which rooms you can enter.

Key Takeaways

• Security is the architecture of trust — designed, not sprinkled on before launch.
• Authentication proves who; authorization decides what. Confusing them is knowing the wrong person very confidently.
• Identities include machines: services, jobs, and integrations need the same trust design as users — location is not identity.
• A JWT is signed, not encrypted: readable by every bearer, trustworthy only after signature, expiry, and audience validation.
• Short access tokens plus protected, rotated refresh tokens: expiry limits damage, rotation shrinks windows.
• Statelessness prices in the revocation problem — logout, offboarding, and compromise all need a designed propagation window (Bonus I, with a badge).
• OAuth2 delegates access; OIDC asserts identity; SSO centralizes both — and makes offboarding one switch instead of fifteen.
• Tenant isolation is enforced from the verified token, never a client-supplied ID.
• Secrets live in an audited, rotating vault — not in code, not in logs, not in env vars alone.
• Encryption is armor, not judgment: it protects data from exposure while authorization decides who belongs.

Interview Lens

Security shows up in interviews as one deceptively small prompt: "How does auth work in your design?"

The weak answer: "We use JWTs." The strong answer names the lifecycle: "Users authenticate at the identity provider; services get short-lived access tokens — fifteen minutes — with rotated refresh tokens; every service validates signature, expiry, and audience; the gateway does coarse-grained checks and forwards identity over a closed trust boundary; business authorization lives in the services; service-to-service uses mTLS with least-privilege accounts; secrets come from the vault; and revocation propagates within one access-token lifetime."

Expect the probes: "How does logout work with JWTs?" — the revocation question, and the differentiator; answering "short expiry plus refresh-token revocation, with a tracked window" beats pretending statelessness has no price. "How do services trust each other?" — mTLS or signed service tokens; volunteering "internal network isn't a trust boundary" wins the room. "How do you isolate tenants?" — from the validated token, enforced on every query, never the request's word. "Where do secrets live?" — vault, least privilege, rotation, audit — four words that signal you've operated this.

Real-World Engineering Lens

Run the credential-age audit. List every token lifetime, API key, and secret with its last-rotation date. The oldest item on that list is your current risk champion.

Scan for secrets in CI. Automated, on every commit — because history remembers what reviewers miss.

Drill the offboarding. Disable a test account at the IdP and time how long its access actually survives across systems. That number is your revocation lag — Bonus I's window, measured in production.

Log security events from day one — logins, denials, secret access, admin actions — with correlation IDs. Future-incident-you is begging.

Separate environments completely. A staging key that opens anything in production isn't a convenience; it's a pre-positioned incident.

What's Next

This article kept ending sentences the same way: log it, audit it, measure the window, watch the credential age.

Trust, it turns out, is only as good as your ability to see it working, and to notice, fast, when it isn't.

Blog 12 — Observability: Designing Systems That Tell You When They Break. Metrics, logs, and traces — the vital signs this series has been collecting since Part 6 (cache hit rates, consumer lag, per-shard p99s, DLQ depths, audit trails) finally assembled into one discipline.

Question for Readers

Right now, in your system: what is the longest-lived credential — a token, an API key, or a secret? When was it last rotated?

If the honest answer is "since launch," congratulations — you've just met your attack surface's favorite door.