Discord: A case study in performance optimization

Hacker News · Feb 14, 2026 · Collected from RSS

Full Article

Justifiable complexity (?)Discord is a permanent, invite-only space where people can hop between voice, video, and text. On the surface, it seems like “just another chat app.” Take a closer look, and you’ll see that it’s really a finely-tuned system that delivers speed, scale, and reliability — the consumer app hat-trick.Every time you send a message, join a voice channel, or watch a stream, Discord has to route the event to the right place, notify tons of clients, and do so fast enough that it feels instantaneous. That’s easy when your server has 50 people. It’s insane when it has 19 million.This is the story of the creative optimizations that keep Discord snappy at scale.Before digging into Discord’s implementation details, we need to understand the architectural pattern upon which it’s built: The Actor Model.Carl Hewitt introduced this idea in his 1973 paper (pdf). Alan Kay embraced it for passing messages in the 80s. Gul Agha formalized its relevance for distributed systems in 1985. Since then, many teams and tools have adopted the model. If you’ve ever read an elaborate sequence diagram or worked in an “event-driven architecture,” you can thank the Actor Model.In shared-memory, multiple threads use the same state, which quickly results in race conditions. You can prevent this by adding a data-access constraint: locks. But locks come with their own bugs, like when multiple threads wait for the other to release the lock, resulting in a permanent freeze (deadlock). As systems grow, these problems become bottlenecks.Adapted from Gul Agha, Actors: A Model of Concurrent Computation in Distributed Systems (93).The Actor Model allows data to be updated more easily in distributed systems.An actor is an agent with a mail address and a behavior. Actors communicate through messages and carry out their actions concurrently. Instead of locks, the Actor Model ensures safe concurrency with communication constraints. The Actor Model can be summarized in four rules:1. Each actor owns its state (no one else can directly mutate it) 2. Actors only communicate through messages 3. Actors process messages one at a time (no race conditions) 4. In response to a message, an actor can: - Change its state - Send messages - Create child actorsFollow the rules and avoid race conditionsHere’s an actor who follows the rules.actor.start(); actor.send(message); // send to another actor.subscribe(s => { ...}); // listen to another actor.getSnapshot(); // cached result from .subscribeAs you can see, actors alone are actually pretty simple, almost pure.As long as you play by the rules, you won’t have race conditions and lock spaghetti. You also get a few other benefits:Location independence. The interface means that you can be confident each actor will behave consistently regardless of its location. Doesn’t matter if one actor is on localhost and another is remote. Two actors can even share the same thread (no thread management!).Fault tolerance. If one actor fails, its manager can revive it or pass its message to an available actor.Scalability. Actors are easy to instantiate, making them compatible with microservices and horizontal scaling.Composability. They encourage atomic over monolithic architecture.How actors can chat asynchronously. Adapted from theserverside.comThere was a quiet period after Hewitt dropped his paper, but adoption has recently taken off in response to our growing data footprint.Here are a few modern examples of the actor model:Video editing software sends each setting change so that they are reflected in the draft immediately.Trading platforms (Robinhood) treat each withdrawal as an isolated actor, which has a function to update an account. Say you have $10 in your account. If you and your wife both try to buy $10 of $GME simultaneously, your app will process the request that arrives first. When the second gets out of the queue, it’ll run the checkBalance statement, see that it’s now $0, and deny it.AI agents. An agent is an actor. It passes messages (prompts), has internal state (context), and spawns other agents (actors). AI Agents are the perfect use case for the pattern, as Hewitt anticipated. His original paper was called “A Universal Modular ACTOR Formalism for Artificial Intelligence,” after all.If you squint past the trendy design and jargon on Akka’s site, it looks like the same pitch Hewitt made five decades ago.Event-driven fabricMulti-step agentsPersisted state through snapshots and change eventsIsolated execution units that maintain their own stateParallel processing without shared bottlenecksUnbounded data flowsThe next time you see these keywords on a landing page, remember to get out your Actor Pattern Buzzword card and yell “Bingo!”Adapted from akka.ioThis concept is relevant on the frontier as well as in the verticals. Cursor’s self-driving codebase experiment took a first-principled journey from unconstrained sharing to a well-defined communication flow between planner and workers. Sounds very similar to actors and managers, doesn’t it?Is it silly that we’re pretending to be pioneers by rejecting the idea of a global state? Yes, but at least we’re not the only ones who need to rediscover old problems: the agents in Cursor’s experiment also tried and failed to make locks work.Agents held locks for too long, forgot to release them, tried to lock or unlock when it was illegal to, and in general didn’t understand the significance of holding a lock on the coordination file. — Wilson Lin @ CursorThankfully, the Actor Pattern still works regardless of our willingness to recognize it.It’s really cool that the Actor Pattern has gained relevance over the decades. But the model isn’t without trade-offs. Things become more complicated when you add production necessities, such as a manager, callbacks, promises, initial state, and recovery. The ease of composition will inevitably lead some teams into microservice hell, where they’ll get lost in boilerplate and hairball graphs.Oops, you got excited and now have 4,000 microservices.Debugging dataflow bugs can be harder. Although each actor has well-isolated logging, it can be harder to trace a bug across multiple services.Price. Agents that create more agents is a dream for the engineering team. If that process isn’t constrained, however, it’ll turn into a nightmare for the finance team.Finally, implementing the actor model in a big org requires education. Pure functions, state machines, event-driven architecture — these are unfamiliar concepts to many. It took me days of research before I “got it.” Many orgs won’t want to dedicate the time to get everyone thinking in a new paradigm, so they’ll fall back to their monolithic habits.Thankfully, the industry has started bridging the gap between the usefulness of this Actor Model and its adoption complexity by creating languages and frameworks. These preserve the actors’ tenets while making them easier to implement.They’re all actorsThe actor model makes it easier to avoid locks and race conditions in distributed systems. It does this by standardizing communication and data access.Letting too many things share the same data and chat freely leads to chaos. Think: startup engineer who has to handle info from Slack, hallway requests from the PM, email, and standup. That guy is going to overcommit (deadlock), forget (data loss), and struggle to organize (recovery).Using an actor model is like requiring everyone to communicate over email. Everyone follows the rules of SMTP (recipient, subject, body) and can only respond to one email at a time (concurrency). In this system, the communication constraints minimize mistakes and conflicts.All this adds up to a faster, more reliable system at scale. Everyone knows how to talk to each other. They know how to ask for and deliver things. This helps them work autonomously without blocking others.Having an efficient pattern becomes more important the more distributed a system becomes. As Gul predicted in 1985, more time is spent on “communication lags than on primitive transformations of data.” A team that knows that all too well is Discord, which has successfully instrumented the Actor Model to process trillions of messages without data loss or latency.Let’s see how.Everything is an “actor.” Every single Discord server, WebSocket connection, voice call, screenshare, etc... distributed using a consistent hash ring.It’s an incredibly great model for these things. We’ve been able to scale this system from hundreds to hundreds of millions of users with very little changes to the underlying architecture. — Jake, Principal Engineer @ DiscordThanks to Discord’s initial target user, gamers, speed has always been an unquestioned requirement. When a message is sent, others need to see it immediately. When someone joins a voice channel, they should be able to start yapping right away. A delayed message or laggy chat can ruin a match.Discord needed a smooth way to turn plain text/voice data into internal messages and then route them to the correct guild (AKA: Discord server) in real-time.Guilds and users talk over Elixir and WebSockets.Users connect to a WebSocket and spin up an Elixir session process, which then connects to the guild.Each guild has a single Elixir process, which acts as a router for all guild activity.A Rust data service deduplicates API queries before sending them to ScyllaDB.Background communication happens over PubSub. For example, the Elasticsearch worker consumes events, batches with others, and starts indexing.“Fan-out” refers to the act of sending a message to multiple destinations in parallel without requiring a response. This is exactly what Discord needed to implement to make their chat feel real-time. When a user comes online, they connect to a guild, and the guild publishes presence to all other connected sessions. Each guild and connected user has one long-lived Erlang process. The guild’s process keeps track of client sessions and sends updates to them. When a gu