Scaling Live Streaming Announcements with Redis Pub/Sub at Enterprise Scale
Infrastructure · Real-time Systems · 2024
Scale
500K+ concurrent users
Read time
8 min
Challenge
Project story Write the case study article body here.
Solution
Project story Write the case study article body here.
Project story
The Problem Worth Solving
As CTO, I oversaw the design and delivery of a real-time announcement broadcasting system for our live streaming platform — one that needed to push event notifications (host joins, stream starts, countdown tickers, moderation alerts) to hundreds of thousands of concurrent viewers with sub-100ms latency and zero tolerance for message loss during peak traffic windows.
Our previous architecture — polling-based REST endpoints backed by PostgreSQL — collapsed under load. Viewer counts spiked 40x during major broadcast events, latency ballooned to 8–12 seconds, and the announcement module became the single point of failure for the entire streaming experience.
This case study details how we rebuilt the announcement module ground-up using Redis Pub/Sub as the messaging backbone, the architectural trade-offs we navigated, the engineering decisions made at each layer, and the measurable outcomes we achieved in production.
The Redis Pub/Sub Message Flow
We designed a five-layer message propagation model. Each layer has a single responsibility, making the system debuggable, independently scalable, and replaceable.
The key insight: Redis acts as the cross-pod communication bus. Every WebSocket server pod subscribes to all active stream channels on startup. When an announcement is published, Redis instantly fans it out to every subscribed server, each of which then pushes to its own pool of connected WebSocket clients. This eliminates the need for a shared state database in the hot path entirely.
Channel Naming Strategy & Message Schema
We used a deterministic, hierarchical channel namespace to enable pattern-based subscriptions using Redis's PSUBSCRIBE command — allowing a single subscriber to listen to wildcard patterns across thousands of streams without managing individual subscriptions.
// Channel naming convention // stream:{streamId}:announce → general announcements // stream:{streamId}:system → system-level alerts // stream:{streamId}:moderation → mod actions const CHANNEL_PATTERNS = { announce: `stream:${streamId}:announce`, system: `stream:${streamId}:system`, moderation: `stream:${streamId}:moderation`, }; // Message payload schema (JSON) { type: "STREAM_START" | "HOST_JOIN" | "VIEWER_MILESTONE" | "ALERT", streamId: "uuid-v4", payload: { /* typed per announcement type */ }, ts: 1712345678901, // epoch ms ttl: 30000, // client-side display TTL in ms priority: "high" | "normal" | "low" }
Publisher Service (Node.js)
The publisher service is intentionally thin — it validates the announcement payload, serializes to JSON, and fires a Redis PUBLISH command. No business logic lives here. This keeps publish latency consistently under 2ms.
import { createClient } from 'redis'; const publisher = await createClient({ url: process.env.REDIS_URL }) .on('error', (err) => logger.error('Redis publisher error', err)) .connect(); async function publishAnnouncement(streamId, announcement) { const channel = `stream:${streamId}:announce`; const message = JSON.stringify({ ...announcement, ts: Date.now(), streamId, }); const receiverCount = await publisher.publish(channel, message); logger.info({ channel, receiverCount }, 'Announcement published'); return receiverCount; // # of subscribed WS pods that received it }
WebSocket Server — Subscriber + Fan-out
Each WebSocket server pod maintains a dedicated Redis subscriber connection. On receiving a Pub/Sub message, it resolves the target room (mapped 1:1 to a stream channel) and pushes to all locally-held sockets in O(n) time, where n is the number of local viewers on that stream.
import { createClient } from 'redis'; import { WebSocketServer } from 'ws'; const subscriber = await createClient({ url: process.env.REDIS_URL }).connect(); const roomRegistry = new Map(); // streamId → Set<WebSocket> // Subscribe to all stream announcement channels via pattern await subscriber.pSubscribe('stream:*:announce', (message, channel) => { const streamId = channel.split(':')[1]; const room = roomRegistry.get(streamId); if (!room?.size) return; // no local viewers for this stream room.forEach((ws) => { if (ws.readyState === WebSocket.OPEN) { ws.send(message); // raw JSON — no re-serialization } }); }); // Register viewer on connection wss.on('connection', (ws, req) => { const { streamId } = parseQuery(req.url); if (!roomRegistry.has(streamId)) roomRegistry.set(streamId, new Set()); roomRegistry.get(streamId).add(ws); ws.on('close', () => roomRegistry.get(streamId)?.delete(ws)); });
Late-joiner replay via Redis Streams
Critical announcements are also written to a Redis Stream (XADD). On WebSocket connection, clients request the last N messages via XRANGE before subscribing to the live Pub/Sub channel.
Idempotent delivery on reconnect
Every announcement carries a UUID. Clients track the last received ID and request a catch-up diff on reconnect, preventing duplicate rendering via a client-side dedup map.
Heartbeat-based connection health
Server pings every 15s. Clients that miss 2 consecutive pings are disconnected and auto-reconnect with exponential backoff (250ms → 4s), triggering a catch-up fetch.
Redis Sentinel for HA
We run Redis in Sentinel mode with 1 primary + 2 replicas. Automatic failover completes in under 3 seconds, and subscriber clients reconnect and re-subscribe transparently via the ioredis retry strategy.
"Redis Pub/Sub gave us the speed we needed. The real engineering challenge was building the durability and reliability layer on top of it — Redis Streams for persistence, Sentinel for HA, and client-side idempotency for correctness."
— CTO, Enterprise Live Streaming PlatformMeasured Outcomes in Production
After a phased rollout over 6 weeks — dark launch, 5% canary, full production — we observed the following in production monitoring via Datadog and Redis INFO stats:
P99 announcement latency: 47ms
Down from 8–12 seconds under the polling architecture. Measured publisher → client receipt with distributed tracing.
500K+ concurrent users handled
Tested under synthetic load simulation. Redis throughput peaked at 1.2M messages/sec across 12 WebSocket pods during load testing.
Database load dropped 91%
Eliminating announcement polling removed the dominant read pattern from PostgreSQL, improving query performance across all other features.
72% infra cost reduction
Fewer app servers needed due to WebSocket efficiency vs HTTP polling. Redis cluster costs were negligible relative to the compute savings.
Infrastructure · Real-time Systems · 2024