Notification Service

Nobody opens a ticket when a notification arrives. They open one when it arrives twice. Or late. Or at 3 AM. Or never.

Part 13 designed a system whose data stood still — immutable mappings, the cache's favorite meal. Today's system is the opposite: nothing but data in motion. Every order confirmation, OTP, delivery update, password reset, and marketing campaign in a product flows through one service whose entire job is Part 7's machinery — queues, pub/sub, retries, DLQs, delivery guarantees — promoted from concepts to the whole product.

And it carries the highest stakes-to-glamour ratio in system design. Done well, nobody notices. Done badly, users get charged-twice alerts twice, OTPs arrive after the login page times out, and one Diwali campaign delays every password reset in the country.

Same template as Part 13. New physics. Let's walk it.

Why the Notification Service Is the Perfect Second Problem

Part 13 taught the read path. This problem teaches everything Part 13 deliberately kept small.

It's asynchronous to its core. There is no synchronous version of "send 50 million messages a day." Part 7's entire toolkit — producers, brokers, consumers, fan-out, backpressure — stops being background theory and becomes the architecture.

It makes delivery guarantees a product feature. "At-least-once plus idempotency" was a Part 7 concept. Here, it's the difference between one OTP and two, and users notice two.

It forces multi-tenant thinking about traffic classes. An OTP and a newsletter are both "notifications." Treating them the same is the design mistake this article exists to break.

And it's universally relevant. Every product you will ever build sends notifications. Most build this service three times before building it right.

A good architect does not ask only, "Did the message get sent?" A good architect asks, "To which traffic class does this message belong, and what does that class guarantee?"

Step 1 — Clarify: The Questions Before the Boxes

"Design a notification service" hides a dozen products. The questions that separate them: which channels — push, email, SMS, in-app, or all four? What types — transactional (OTP, order updates) and promotional (campaigns), do both flow through us? How fast is "fast"? Is an OTP arriving in 60 seconds a success or an incident? Do users control preferences — channels, categories, quiet hours, unsubscribe? Broadcast — can one event target millions of users at once? Do we track delivery, or just hand messages to providers and hope?

Our assumptions, stated:

Channels:       Push + email + SMS + in-app (all four)
Types:          Transactional AND promotional — with DIFFERENT guarantees
OTP latency:    Seconds — this is the strictest requirement in the system
Preferences:    Yes — channels, categories, quiet hours, unsubscribe (regulatory)
Broadcast:      Yes — campaign events fanning out to millions
Tracking:       Yes — per-notification status, including provider receipts

The second answer is the design's spine: one service, two very different citizens. Hold it.

Step 2 — Requirements: The Building Code

Functional: accept notification requests (single and broadcast); resolve user preferences; render templates; route to channel providers; retry failures; track status; honor unsubscribes and quiet hours.

Non-functional, and notice how they split by type (Bonus B's per-journey discipline):

Transactional (OTP, order, security alerts):
  Delivery latency p99      < 5 seconds end-to-end
  Delivery guarantee        effectively-once — NEVER twice for OTPs/alerts
  Availability              99.95% — this path outranks everything

Promotional (campaigns, digests):
  Delivery window           minutes to hours is fine
  Delivery guarantee        at-least-once; rare duplicates tolerable
  MUST NOT interfere        with transactional latency — ever

Both:
  Preference compliance     unsubscribes honored within minutes (regulatory)
  Retention                 90-day notification log, then archive (Part 8)
  Privacy                   no sensitive content in logs; PII minimized

That "MUST NOT interfere" line is an NFR most teams discover only after the incident. We're writing it down first.

Step 3 — Estimate: The Numbers

Bonus A's method, applied fresh:

Assume: 10M DAU, ~5 notifications each per day

Volume:     50M notifications/day
            ÷ ~100,000 sec  ≈ 500/sec average
            × 5 peak factor ≈ 2,500/sec at peak

Broadcast:  one campaign to 5M users = 5M messages
            injected in minutes — a BURST 100× the average
            (Part 7's load-leveling is not optional here)

Storage:    50M/day × ~1 KB status record × 90 days ≈ 4.5 TB
            → real lifecycle territory (Part 8), unlike Part 13's ½ TB

Ratio:      write-heavy — the inverse of Part 13

The verdict the numbers deliver: Part 13 was a read problem with a light write path. This is a write-and-deliver problem whose bursts dwarf its average — the queue isn't an optimization, it's the load-bearing wall. And the storage story is ten times bigger, so retention rules ship on day one.

Step 4 — Design: The High-Level Architecture

Producers (order svc, auth svc, campaign engine…)
        │  publish NotificationRequested events (Part 7)
        ▼
┌─ Ingestion API ──────────────────────────────┐
│  validate • authenticate (Part 11) •         │
│  assign idempotency key • classify priority  │
└──────┬───────────────────────────┬───────────┘
       ▼                           ▼
  TRANSACTIONAL LANE          PROMOTIONAL LANE
  (its own queues,            (its own queues,
   its own workers)            its own workers)
       │                           │
       ▼                           ▼
┌─ Notification Pipeline (per lane) ───────────┐
│  1. Preference check   (may this be sent?)   │
│  2. Channel resolution (where?)              │
│  3. Template rendering (what exactly?)       │
│  4. Dedup check        (already sent?)       │
└──────┬───────────────────────────────────────┘
       ▼
  Provider Adapters:  Push ─ Email ─ SMS ─ In-app
       │                    (retries • failover • rate limits per provider)
       ▼
  Status Store  ◀── provider delivery receipts (webhooks)
       │
       └──▶ events → analytics (async, Bonus H)

The one decision that defines this design: separate lanes by traffic class.

Transactional and promotional notifications share code, templates, and providers, and share no queues and no workers. A 5-million-message campaign fills the promotional lane's backlog for an hour, and the OTP lane never feels it. This is the bulkhead idea (Part 22 will formalize it): the blast radius of the big, deferrable traffic is walled off from the small, sacred traffic.

Skip this separation and the design fails its own NFR on the first big campaign — the marketing blast buries the password resets, and "notifications are down" trends on the day you least want it.

Store map (Bonus H's discipline): the status store is the source of truth for what was requested and what happened to it. Queues are in-flight work, not truth. The dedup store and analytics are derived. Templates and preferences each have one owning service.

Deep Dive 1: The Pipeline — From Event to Provider

Walk one notification through.

Preference check. May this message go to this user, on this channel, right now? Unsubscribes, category opt-outs, and quiet hours live here — quiet hours meaning some messages aren't dropped but deferred (a scheduled-delivery queue: same broker machinery, plus a clock).

And the question that reveals design maturity: what if the preference service is down? There's no single answer — there's a per-type answer, and it's this article's version of Part 13's 301-vs-302. Promotional: fail closed. When in doubt, don't send. A missed newsletter costs nothing; an unsubscribed user receiving a campaign is a complaint and possibly a regulatory violation. Transactional: fail open. When in doubt, send. A delayed OTP locks a user out of their account; nobody has ever unsubscribed from their own password reset.

Same failure. Opposite correct behaviors. Written down per type, in advance (Bonus C: a business trade-off, decided by the business, encoded by us).

Channel resolution. OTP → SMS + push. Order update → push, fall back to email if no device token. Campaign → per user preference. The routing table is configuration, not code — it changes weekly; deployments shouldn't.

Template rendering. Message templates with variables ("Your order {id} has shipped"), versioned like the contracts they are (Part 10's discipline — a renamed template variable is a breaking change that fails silently at render time). Rendered in the pipeline, not by producers — producers send facts; the notification service owns presentation.

Provider adapters. Each channel talks to external providers — email services, SMS gateways, push platforms. Every adapter carries the same kit: per-provider rate limits (providers throttle you; exceed the limit and they start dropping), timeouts, retries with backoff (Part 7, Bonus E's budgets), failover to a secondary provider when the primary's error rate spikes, and credentials from the vault (Part 11 — provider API keys are exactly the secrets that end up in code).

Deep Dive 2: Effectively-Once — The Deduplication Design

The requirement said it plainly: an OTP must never arrive twice. Part 7 taught why that's hard: at-least-once delivery means redelivery is normal — a worker that crashed after calling the SMS gateway but before acknowledging will cause the message to be processed again.

The broker cannot fix this. Only the business logic knows what "already sent" means. So we build it.

The idempotency key: event_id + user_id + channel. Not random — derived from the business fact, so any retry, any redelivery, any producer-side double-publish computes the same key.

The dedup flow:

Worker picks up message
   → compute key: evt_8841:user_5521:sms
   → dedup store: atomic check-and-claim, with TTL

        already claimed → acknowledge, do nothing  ← the duplicate dies here
        claimed now     → call provider → record status → acknowledge

Three design notes with teeth. The claim happens before the provider call — claiming after sending recreates the exact crash window we're closing. The TTL is a judgment call: long enough to cover any realistic retry horizon (hours), short enough that the store doesn't grow forever. Expired keys are why this is effectively-once, not mathematically exactly-once, and why the transactional lane's retry horizon is kept short. And the crash-between-claim-and-send case leaves a claimed-but-unsent notification — the status store catches it: a reconciliation sweep finds claimed records with no sent status and re-issues with the same key after verifying no provider receipt exists. Rare, bounded, owned.

And the sentence that keeps the whole section honest: for promotional traffic, we don't pay this cost. At-least-once is fine for a newsletter; the dedup machinery guards the lane where duplicates hurt. Guarantees are per-type, like everything else in this system.

Failure Analysis: Where This Design Bends and Breaks

An email provider goes down. The adapter's error rate spikes → failover routes to the secondary provider; the queue absorbs the switchover wobble. No secondary? Then the queue simply holds — Part 7's temporal decoupling — and drains on recovery. The metric that matters: lane lag, alerting before users do.

A 5M-user campaign lands at peak. The promotional lane's backlog explodes — by design. Workers drain it at their sustainable pace (drain time = backlog ÷ spare capacity, Bonus A's formula, now on a dashboard). The transactional lane, in its own bulkhead, never notices. This "failure" is the architecture succeeding.

A worker crashes mid-send. Redelivery happens (at-least-once); the dedup claim eats the duplicate. The OTP arrives once. This is the entire dedup section, earning its keep in one sentence.

The preference service fails. Per-type behavior, decided in advance: promotional fails closed, transactional fails open. The incident becomes a footnote instead of a debate.

A poison message — a template that crashes rendering. Three retries, then the DLQ, with an alert (Part 7's rule: a DLQ nobody monitors is a slower way to lose messages). One malformed campaign must never wedge the pipeline behind it.

Provider receipts flood back. Five million sends produce five million webhooks. The receipt endpoint is itself a high-throughput ingestion path — it gets its own queue, and receipts update the status store asynchronously. (The system that sends bursts must be ready to receive them.)

The dedup store dies. The sharpest trade-off in the design: fail open (risk duplicates) or fail closed (risk delays)? For OTPs — where a duplicate confuses but a missing OTP locks users out — most teams choose degraded-open with loud alerts. Decide it now; write it down (Bonus C).

Every failure: named, bounded, answered in advance. Part 13's closing rule holds — the system is designed when its failures are understood.

Security and Abuse: The Part Everyone Skips

A notification service holds a dangerous combination: everyone's contact details, and the power to message them at scale. Design for it (defensively, per Part 11): producer authentication and authorization (only registered services may request notifications, each scoped to its own templates and types — Part 11's service identity; the campaign engine cannot send OTPs); content minimization (push previews render on lock screens; sensitive details stay out of previews and out of logs — the status store records that a message was sent, not its rendered secrets); template injection defense (user-supplied variables are data, never markup; templates are authored by owners and reviewed like code); unsubscribe integrity (one-click, honored within minutes, propagated to every lane — the regulatory NFR made mechanical; suppression lists checked at the preference step, always); per-user rate limits (no user receives 400 notifications because a producer looped; the pipeline is the last line of defense against upstream bugs); provider credentials in the vault (Part 11), rotated (an SMS gateway key is a spam cannon with your sender reputation attached); and an audit trail (who configured which campaign to which audience, forever — Parts 11–12).

The Observability Plan

Part 12's table, applied — per lane, always per lane:

SLO:          99.95% of transactional notifications delivered < 5s
Lanes:        queue depth, consumer lag, drain-time projection — PER LANE
Pipeline:     preference-check latency, render failures, dedup hit rate
Providers:    per-provider success rate, latency, throttle responses,
              failover activations
Delivery:     sent vs delivered vs bounced (from receipts), per channel
DLQ:          size per lane — alert on transactional DLQ > 0
Business:     OTP end-to-end p99 (the number that matters),
              unsubscribe latency, campaign completion time
Synthetic:    a robot user receiving a test OTP every minute, per channel

The dedup hit rate deserves a highlight: a rising rate means duplicates are being caught, which means something upstream started retrying more. It's the smoke detector for a fire elsewhere.

What Interviewers Actually Probe

"How do you guarantee an OTP is sent exactly once?" — the honest answer wins: exactly-once is a myth end-to-end; at-least-once delivery plus idempotent claim before the provider call equals effectively-once. Naming the claim-before-send ordering is the differentiator. "A marketing campaign delays OTPs — what went wrong?" — shared lanes; the fix is bulkheaded queues and workers per traffic class; candidates who volunteer this separation before the probe skip the trap entirely. "The email provider is down." — failover, queue absorption, lane lag alerting, and the follow-up "how do you detect it?" (per-provider error rate, not global averages — Bonus G's lesson). "User says they unsubscribed but still got an email." — propagation lag between preference write and pipeline read (Bonus I, between stores) — name the window, its monitor, and the suppression-list check placement. "How do you handle 5M sends in ten minutes?" — drain-time arithmetic out loud (Bonus A), provider rate limits as the real ceiling, and the receipts flood coming back. "What's in your dedup key?" — derived from the business fact, never random: event + user + channel, and why each part is there.

What We Deliberately Didn't Build

A campaign-authoring product — segmentation, A/B testing, scheduling UIs: a whole product upstream of us, we accept its events. In-house channel delivery — running your own SMTP fleet or SMS gateway layer is a company, not a component; providers plus failover is the sane default (Bonus C: a budget-and-muscles call). ML send-time optimization — "notify each user at their favorite hour" is real and valuable, after the reliability core exists. Cross-device read-sync for in-app ("mark read everywhere") — Part 15's real-time machinery is the honest home for it. Multi-region active-active — same verdict as Part 13, same trigger discipline.

Common Mistakes Engineers Make with This Problem

One queue for everything — the campaign buries the OTP, the design fails its own NFR on day one. Random idempotency keys — a key not derived from the business fact dedupes nothing, every retry mints a "new" message. Claiming after sending — the crash window this whole design exists to close, reopened. Retrying without backoff into a throttling provider — Bonus E's storm, aimed at a partner who will rate-limit you harder for it. No DLQ per lane — one poison campaign message wedges everything behind it. Treating provider handoff as delivery — "sent" is your claim, "delivered" is the receipt's; track both or lie to your dashboards. Preference check without a failure policy — the fail-open/fail-closed decision, made at 2 AM by whoever's on call. Sensitive content in logs and previews — Part 11's rule, broken at the highest-volume writer in your estate. No per-user rate limit — the upstream loop that texts one customer 400 times. Ignoring the receipts path — built to send 2,500/sec, toppled by 2,500/sec of webhooks coming back. Same guarantees for every type — paying the dedup tax on newsletters, skipping it on OTPs. No drain-time dashboard — a backlog without a projected recovery time is a status page nobody can write.

From My Journey: An Architectural Lesson

Notification systems tend to enter a codebase as a single innocent function — send an email on signup. Then a push notification on order updates. Then SMS for OTPs. Then a campaign tool. Each addition small, each one reasonable.

And somewhere along the way, without a design review ever being called, the product has grown a reliability-critical distributed system — one that holds every user's contact details, faces external providers with their own moods and limits, and carries traffic ranging from "must arrive in seconds, exactly once" to "sometime today is fine."

The realisation that reframes it: the hard part was never sending messages. Providers make sending trivial. The hard parts are everything around the send — which messages must never mix lanes, which must never happen twice, what happens when the checker is down, and how five million receipts come home. A notification service is a delivery-guarantees product wearing a messaging costume.

And the discipline that fell out: guarantees, failure behavior, and cost are decided per traffic type, never globally. The OTP and the newsletter share a pipeline and nothing else.

The lesson that stuck: a notification system is not judged by how many messages it sends. It is judged by how few wrong ones it sends.

Key Takeaways

• Notifications are Part 7's machinery promoted to a product: queues, fan-out, retries, DLQs, and guarantees — as the feature set.
• The defining decision: bulkheaded lanes per traffic class. Transactional and promotional share code and nothing else.
• NFRs split by type: seconds-and-never-twice for OTPs; hours-and-occasionally-twice for campaigns.
• Effectively-once = at-least-once delivery + an idempotency claim, derived from the business fact, taken before the provider call.
• The preference-service failure policy is per-type: promotional fails closed, transactional fails open — decided in advance.
• Providers are external dependencies with moods: per-provider rate limits, backoff, failover, and vault-kept credentials.
• "Sent" is your claim; "delivered" is the receipt's — and the receipts flood is its own ingestion system.
• The status store is the source of truth; queues are in-flight work; dedup and analytics are derived (Bonus H's map, applied).
• Write-heavy, burst-dominated, 4.5 TB of logs — the estimation verdict is the inverse of Part 13's, and the design followed it.

Interview Lens

The winning shape, same as Part 13: clarify the traffic types first — the moment you say "OTPs and campaigns get different guarantees and different lanes," the interview changes tone, because you've named the trap before stepping near it.

Then let the numbers declare the problem ("write-heavy, burst-dominated — my complexity budget goes to lanes, dedup, and provider resilience"), design the pipeline, and volunteer the failure walk — provider down, campaign burst, crash-mid-send, preference service dark — with the per-type answers attached.

The meta-skill this problem tests: guarantee literacy. Every probe is secretly asking whether you know that "exactly once" is negotiated, not declared, and that the negotiation happens per traffic type, in the design doc, before the incident.

Real-World Engineering Lens

Audit your lanes today. If OTPs and campaigns share a queue anywhere in your estate, you have a pre-scheduled incident with a marketing calendar attached.

Check your dedup keys. Random UUID per attempt equals no dedup at all. Derived from the business fact, or it doesn't count.

Write the two failure policies — preference-service-down and dedup-store-down — per traffic type, in the design doc, this week.

Put drain time on the dashboard, not just queue depth. "Backlog: 2M" frightens; "drains in 14 minutes" informs.

Test the unsubscribe path end-to-end quarterly. It's the one flow where a bug is a regulator's business, not just a user's.

What's Next

We've mastered messages that travel one way — service to user, seconds to hours, fire and track.

The next system removes every comfortable assumption at once: both ends are humans, both ends are waiting, the connection stays open, and "eventually" is measured in milliseconds.

Blog 15 — Chat Application: WebSockets, Message Ordering, and Presence. Real-time delivery, per-conversation ordering, online/offline presence, and the question that breaks naive designs: what does "delivered" mean when the recipient's phone just went through a tunnel?

Question for Readers

Find one notification flow in your product and answer three questions: does it share a lane with traffic of a different priority? What exactly is its idempotency key derived from? And if the preference check failed right now, does it send, or stay silent?

If any answer is "not sure," you've found the incident before it found you.