Scaling Out SignalR: In-Memory State Behind a Load Balancer

A few years ago I was the architect on a live collaboration tool. Multiple users joined the same session, worked over shared mutable state for hours, and everything traveled through SignalR. The product grew and we had over 10.000 simultaneous users, and we decided to put more instances behind the load balancer.

Then the tickets started. Users inside the same session could not see each other. In all the meetings, there were reports of users being isolated or in smaller groups than they should have been. Nothing crashed and nothing logged an error. The system was healthy by every measure we had.

Put side by side, the reports had a shape: the groups users saw were always subsets of their real session, never wrong ones, and which subset you got depended on who you asked. That shape pointed at only one place: our presence tracking lived in an in-memory dictionary, and after the scale-out it got distributed into different servers.

Every server had its own reality

The design was reasonable when it was written. When we had only one server, having one dictionary mapping connections to users made sense.

But scaling it out partitioned the state:

The state partitioned. Each instance kept its own private dictionary. A user who landed on server two was written into server two’s memory and nowhere else. Asking “who is in this session” returned whoever happened to share your server. SignalR’s Clients.All means all clients connected to this process. Without a backplane, a presence update raised on server two never reached the browsers attached to the other servers.

A user did not get an incomplete list once and recover. The list stayed wrong in the same way for as long as the connection lived, and reconnecting did not help, because the balancer pinned you right back to the same server.

Sticky sessions solve a different problem

We did have sticky sessions. New connections were spread across the servers and then pinned, so each user stayed on one server for as long as they were connected. But affinity binds a client to a server, and the thing our state was about was not a client. It was a session: a group of people working together for hours. Nothing pinned the members of one session to the same server, so the balancer scattered them across the whole pool, and each dictionary held only the fragment of the group that happened to land on it. The affinity was real and it was doing its job. It was doing it at the wrong granularity.

Why the development environment did not catch the issue

We did not catch this in development because it was not catchable there. We had one dev server. Every client in every test landed in the same process and shared the same dictionary. Staging was topologically identical to production: multiple instances behind the same balancer with the same affinity. It still could not catch this. One tester on one computer is one client for the balancer to pin, and therefore one server, so every simulated user in the test shared a process exactly as they had on the single-instance dev box. What staging was missing was never more servers. It was more clients. The test suite passed because the code was, on a single instance, genuinely correct, and we failed to produce the same conditions real users had in production. This was a serious point in our post-mortem, and out of it came a new checklist we used from then on to evaluate every infrastructure change before deploying it. The incident also became a lecture: I walked the whole engineering team through it as a distributed-systems teaching moment, so the lesson would not stay locked inside the people who debugged it.

The bug did not exist until the new instances came up in production. It was created by the deployment topology, retroactively, in code nobody had touched for months.

That is the part worth generalizing. Every piece of state you keep in process memory carries an invisible assumption: all traffic sees this memory. Nothing in the type system records that assumption, and tests do not necessarily fail when it stops holding. The change that broke us did not touch application code at all. It was capacity work, adding instances, the kind of change that reads as an ops ticket.

The part that was on me

I greenlighted that change. The dev tests passed, the rollout plan looked clean, and I did not ask the one question that mattered: how does the infrastructure change affect the code it is supporting?

Reviewing the scale-out as a design change, not a capacity change, was exactly my job, and I treated it as routine instead. The developers had implemented the tracker correctly for the system it was born into, and the tests honestly reported that it worked. The process had no step where “what does this code mean on N machines” would ever be asked.

Rebuilding the bug so you can run it

I could not show you the original system, so I rebuilt the failure in miniature: signalr-redis-backplane-demo, a docker-compose stack with three ASP.NET Core instances behind round-robin nginx, a presence hub, and a page that shows two things — which instance your tab landed on, and who it thinks is online. The infrastructure has its own walkthroughs in the repo (the nginx configuration and the compose topology) so this article can stay about the bug.

Two differences from production are deliberate, and both come from the same constraint. Production had affinity and thousands of separate clients for the balancer to scatter. A laptop has one client, and every tab of one browser looks the same to the balancer, so affinity in the repro would pin all of them to one instance and hide the bug. The repro therefore drops affinity, and must skip SignalR’s negotiate step as a consequence, since the default handshake is what needs the affinity in the first place; a tab is then one request that lands on one instance. It also collapses the collaborative session into one global room; the production fix stayed at the session’s granularity, with presence keyed per session and updates sent to that session’s group rather than to every connected client. The invariant it demonstrates is the same: the presence list lives inside one process, and there is more than one process.

The entire lesson lives in one interface with two implementations:

public interface IPresenceTracker
{
    Task AddAsync(string connectionId, string name);
    Task RemoveAsync(string connectionId);
    Task<IEnumerable<string>> GetAllAsync();
}

The broken implementation is the one we shipped, reduced to its essence:

// THE BUG: this dictionary lives inside one process. On a single instance it is
// perfectly correct. Behind a round-robin load balancer each of the three
// instances keeps its own private copy, so a user only ever sees the users who
// happened to land on the same instance. Nothing crashes and nothing logs an
// error -- the app just quietly shows everyone a different reality.
public class InMemoryPresenceTracker : IPresenceTracker
{
    private readonly ConcurrentDictionary<string, string> _users = new();

    public Task AddAsync(string connectionId, string name)
    {
        _users[connectionId] = name;
        return Task.CompletedTask;
    }

    public Task RemoveAsync(string connectionId)
    {
        _users.TryRemove(connectionId, out _);
        return Task.CompletedTask;
    }

    public Task<IEnumerable<string>> GetAllAsync()
    {
        return Task.FromResult<IEnumerable<string>>(_users.Values.ToList());
    }
}

Run the broken mode and open a few tabs:

docker compose up --build
# open http://localhost:8080 in six tabs, join with different names

Each tab tells you which instance it landed on. Tabs on different instances show different user lists, live, with no errors anywhere. The tabs make the bug visible; the test/ directory in the repo makes it objective, connecting six scripted SignalR clients through nginx and reporting what each one saw. This is the output of a run:

"instanceSpread": {
  "app2": ["user1", "user4"],
  "app3": ["user2", "user5"],
  "app1": ["user3", "user6"]
},
"distinctFinalLists": [
  ["user1", "user4"],
  ["user2", "user5"],
  ["user3", "user6"]
],
"allAgree": false

Six users, three disjoint answers. This was what hit us in production, with over ten thousand concurrent users and no error logs. That afternoon was mayhem.

The fix

We resolved it by moving the state out of the process and into Redis. There are two separate problems, and each needs its own fix:

The broadcasts need a backplane. AddStackExchangeRedis makes every instance relay its Clients.All sends through Redis pub/sub, so a presence update raised on one server reaches browsers attached to all of them.

The state needs to move out. The backplane does nothing for the dictionary. If the data itself stays per-process, every instance still answers “who is online” from its private copy. It just broadcasts its wrong answer more widely. The tracker has to read and write shared storage.

In the repro, both fixes together are this:

if (mode == "redis")
{
    // With abortConnect=false the multiplexer starts even if Redis is not up
    // yet and connects in the background.
    IConnectionMultiplexer redis =
        await ConnectionMultiplexer.ConnectAsync(redisConnection + ",abortConnect=false");
    builder.Services.AddSingleton(redis);

    // The backplane relays Clients.All broadcasts between instances...
    signalR.AddStackExchangeRedis(options => options.ConnectionFactory = _ => Task.FromResult(redis));
    // ...and the shared tracker makes every instance agree on who is online.
    builder.Services.AddSingleton<IPresenceTracker, RedisPresenceTracker>();
}

public class RedisPresenceTracker : IPresenceTracker
{
    private const string Key = "presence";
    private readonly IDatabase _db;

    public RedisPresenceTracker(IConnectionMultiplexer redis)
    {
        _db = redis.GetDatabase();
    }

    public Task AddAsync(string connectionId, string name)
    {
        return _db.HashSetAsync(Key, connectionId, name);
    }

    public Task RemoveAsync(string connectionId)
    {
        return _db.HashDeleteAsync(Key, connectionId);
    }

    public async Task<IEnumerable<string>> GetAllAsync()
    {
        var entries = await _db.HashGetAllAsync(Key);
        return entries.Select(e => e.Value.ToString()).ToList();
    }
}

Same stack, same nginx, same three instances:

docker compose down
PRESENCE_MODE=redis docker compose up

Open the same tabs again. Every tab now shows every user, no matter which instance it landed on.

Where this leaves you

The dictionary was the right call when we had only one server. The rule it broke only came into force later: a process can scale horizontally only while the state it holds is either a disposable copy of truth kept somewhere else, or data scoped to a single connection. Everything else makes the process the authority for that data. Our dictionary was the authority for “who is online”, so adding instances quietly created many authorities where the design assumed one, partitioned by server when the thing they were authoritative about was a session. Nothing in the system had ever been asked to notice the difference.

The fix has real costs, and it is worth naming them instead of presenting Redis as a happy ending. Every join and leave now pays a network round trip on what used to be a dictionary lookup. Redis becomes a dependency to size, monitor, and make redundant, because centralizing the truth also centralizes the failure domain. Broadcasts cross the network twice, once in to the backplane and once out to every instance. Shared presence entries outlive a hard-killed instance unless something sweeps them, which is why real systems pair this pattern with a TTL or a heartbeat. And the backplane itself is pub/sub with at-most-once delivery, so a dropped message leaves a client holding a stale list until the next event; the fix does not eliminate silent divergence so much as move it behind a dependency you can monitor. If you would rather buy than operate, that is what a managed service like Azure SignalR Service gives you for the connection and broadcast half. The state still has to live somewhere shared.

Sticky sessions are not the shortcut past any of this; we already had them and kept them afterwards. Every pin is also drawn again on the next deploy or failover, so even the per-client stability it gives you is only as durable as the topology.

I will leave you with one question. For every static, singleton, and cache in a system that runs on more than one instance: is this memory the authority for something, or a copy of it?