Design Twitch

You're asked to design a live streaming platform. A broadcaster goes live from a home connection, and within a few seconds hundreds of thousands of viewers are watching at whatever quality their connection supports. Viewers chat alongside the stream and expect the broadcaster to react to their messages, so the delay between something happening on camera and appearing on screen has to stay small. When the stream ends, it becomes a replayable recording.

Live content does not exist until viewers need it. We must ingest, transcode, package, and deliver every segment within a budget of a few seconds for the full broadcast. Let's write down that latency budget before choosing protocols or segment sizes.

Clarify the requirements

• How much delay is acceptable between the broadcaster's camera and the viewer's screen? This is the first question and it changes everything. Two to five seconds supports chat interaction; thirty seconds does not; sub-second requires a different protocol entirely.
• How large can one stream get? A few hundred thousand concurrent viewers for a large event, which is a fan-out problem rather than a storage one.
• How many concurrent broadcasters? Tens of thousands, and most of them have very few viewers, which is a distribution that matters for cost.
• Is chat in scope? Yes, and its synchronization with video is a good deep dive.
• Is the stream saved for replay afterwards? Usually, which means recording while the broadcast is still going out live.

Assume: two to five seconds of latency, streams up to a few hundred thousand viewers, tens of thousands of concurrent broadcasters, chat in scope, and streams saved for later replay.

Back-of-envelope numbers

• Broadcasters: $50\text{k}$ live concurrently
• Viewers: $5\text{M}$ concurrent, extremely unevenly distributed, since the top few streams hold a large share
• Ingest: $50\text{k} \times 6\text{ Mbps} = 300\text{ Gbps}$ inbound, which is modest
• Transcode: every live stream needs its full set of quality variants, called the bitrate ladder, with one encoded version per resolution and bitrate the player might request, produced in real time, so $50\text{k}$ concurrent real-time encodes is the dominant compute cost
• Egress: $5\text{M} \times 3\text{ Mbps} = 15\text{ Tbps}$
• Chat on a large stream: $100\text{k viewers} \times 1 \text{ message per few minutes} \approx 500\text{ msg/sec}$ inbound, but delivering each to $100\text{k}$ viewers is $50\text{M deliveries/sec}$

Fifty thousand simultaneous real-time encodes dominate compute cost. Chat fan-out on one large stream may exceed the video path in message volume. Let's treat video and chat as separate systems with separate scaling strategies.

High-level architecture

Components

1. Ingest edge. The nearest server accepting the broadcaster's uploaded stream, using a protocol built for uploading video such as RTMP or SRT.
2. Live transcoder. Produces the bitrate ladder in real time, the system's dominant compute cost.
3. Packager. Cuts each rendition into short segments and keeps the index of available segments up to date.
4. CDN edge. Serves segments to viewers; the only component that sees viewer-scale traffic.
5. Replay storage. Segments kept after the broadcast so the stream can be watched later.
6. Chat service. Accepts messages and applies moderation.
7. Chat fan-out. Delivers to every connected viewer of that channel.

Deep dive 1: where the latency actually goes

Define a latency budget before choosing the architecture. Between the broadcaster's camera and the viewer's screen, we spend time in:

• Encoder buffer on the broadcaster's machine. Typically 1 to 2 seconds, and largely out of your control.
• Network to the ingest point. Small, if ingest is geographically close, which is why ingest runs at many points of presence rather than one region.
• Transcode. Roughly the duration of one segment, since encoding can't emit a segment before it has one.
• Segment duration and player buffer. The largest and most controllable term. A player holding three segments of four seconds each is twelve seconds behind the broadcaster before anything else is counted.
• CDN propagation. Small, but nonzero on a cache miss.

The dominant lever is segment duration, and it's a direct tradeoff. Short segments cut latency and multiply request overhead and manifest churn; long segments are efficient and leave viewers far further behind the broadcaster. Two seconds is a common compromise, and low-latency HLS improves on it by delivering partial segments as they're produced, so the player can begin before a full segment exists.

When subsecond latency is required, such as for an auction or interactive game show, segmented protocols are not enough. We can use WebRTC selectively. It delivers subsecond latency but costs much more per viewer because each viewer has a peer connection instead of requesting cacheable files.

Deep dive 2: transcoding in real time

Live transcoding is unlike on-demand transcoding in the way that matters most: it cannot be parallelized across time. On-demand encoding splits a video into a thousand chunks and encodes them simultaneously. A live stream arrives at one second per second, so a single stream's ladder is limited by real-time encoding speed.

Parallelism comes from renditions rather than time. We can encode ladder variants concurrently from the same decoded frames, but each stream still advances one second per second. With fifty thousand broadcasters, this stage dominates compute cost.

Control the cost in three ways:

• Don't transcode everything. Most broadcasters have almost no viewers. Passing their stream through at source quality only, and transcoding a full ladder on demand when viewership crosses a threshold, saves an enormous fraction of the total cost.
• Encode fewer renditions for smaller streams. A stream with fifty viewers doesn't need six variants; three cover almost all devices.
• Use hardware encoders. GPU or dedicated ASIC encoding is dramatically cheaper per stream than software at this volume, with a small quality cost that's acceptable for live video.

Ingest for live video should be geographically distributed, because a broadcaster on a home connection has an unreliable uplink. A long network path to the ingest point means packet loss, and that loss degrades the stream for every viewer watching it. Accepting the broadcaster's upload at the nearest ingest location keeps that fragile first hop short.

Deep dive 3: fanning out to viewers

Delivery to viewers is the part that looks hardest and is in fact the most tractable, because a live stream's segments are still just files.

The manifest has to be rewritten continuously. A manifest is the small index file telling a player which segments exist and where to fetch them. For on-demand video the manifest is written once and lists every segment in the finished video. A live stream has no end yet, so its manifest lists only the most recent segments, and that window slides forward as new segments are produced and old ones fall off. The player re-fetches the manifest every second or two to find out what's new.

The segments themselves never change once written, so they cache exactly like on-demand ones. The manifest is the opposite: it changes every couple of seconds, so it can't be cached the same way. Two things resolve that:

• A short cache lifetime on the manifest. Keep it shorter than the segment duration so viewers learn about new segments promptly.
• A shield tier in front of the origin. A hundred thousand viewers re-fetching the manifest every two seconds is fifty thousand requests per second reaching the origin, if every CDN edge fetches it independently. Putting one intermediate cache layer between the edges and the origin collapses that to a single request per interval per shield.

Popularity is extremely skewed, and it's worth designing to. The top streams are watched by hundreds of thousands and are perfectly cached at every edge; the long tail has a handful of viewers each and gets almost no cache benefit. Pre-warming edges for a scheduled large event, and accepting near-origin serving for tiny streams, uses capacity in proportion to where it's needed.

Saving the stream for replay comes almost free. The same segments the packager produced for live delivery are retained rather than expired, and the sliding manifest is replaced with a complete one listing every segment when the stream ends. No re-encoding is required, which is a satisfying property to point out.

Deep dive 4: chat, and keeping it aligned with video

Chat is a separate system that shares a channel identity with the video, and its scale is easy to underestimate: on a large stream, message deliveries outnumber anything on the video path.

Fan-out is per-channel, not per-user. Viewers connect over WebSocket and subscribe to a channel; a message published to that channel is delivered to its subscribers by a fan-out tier that shards by channel. A very large channel spans multiple fan-out nodes, all subscribed to the same message stream.

Three techniques carry most of the load:

• Batch delivery. Sending messages in small time-windowed batches rather than individually collapses per-message overhead, and a few hundred milliseconds of grouping is imperceptible in a chat scrolling too fast to read.
• Sample above a threshold. On a channel with a hundred thousand viewers, chat is unreadable regardless. Delivering a subset preserves the experience at a fraction of the cost, and viewers cannot tell.
• Prioritize what matters. Moderators, subscribers, and the broadcaster's own messages are always delivered; general chat is what gets sampled.

Moderation has to be synchronous with delivery, since a deleted message that has already reached a hundred thousand clients cannot be recalled. Automated filtering runs before fan-out, with human moderator actions applying to subsequent messages.

Timestamp chat messages against the stream's timeline. Viewers on different quality levels and different buffer depths are seconds apart in the video, but chat reaches everyone at the same moment, so a viewer running several seconds behind reads reactions to something they haven't seen yet. Tagging each message with the point in the stream it was sent at, and having the client hold it until playback reaches that point, fixes it. It's a small change that noticeably improves the experience, and raising it shows you're thinking about the product rather than only the pipeline.

Common pitfalls

• No stated latency budget. "Low latency" without a number means the segment duration decision has nothing to anchor it.
• Proposing WebRTC for everything. It solves sub-second latency at a cost per viewer that doesn't scale to a large audience.
• Transcoding a full ladder for every broadcaster. Most streams have almost no viewers, and this is where the compute budget disappears.
• Delivering every chat message individually on a large stream. Deliveries, not messages, are the scaling problem.
• Ignoring chat-to-video alignment. Viewers read reactions to moments they haven't reached.

Leveling signals

Related lessons

Design YouTubeHard

Ingest and transcode continuous user uploads, then deliver them worldwide.

Design SlackHard

Deliver messages to large channels, where one message fans out to many thousands of members.