WebSocket Reconnects, Ordering & Fan-Out at Scale

WebSocket Reconnect, Ordering, and Fan-Out

A WebSocket demo works on the first try: open a connection, send messages, done. Production is different. Mobile networks drop silently, load balancers close idle connections, laptops sleep, servers restart during deploys, and tokens expire mid-session. Each disconnect raises questions the demo never answered: which messages did the client miss? Will it see some twice, or out of order? What happens when 50,000 clients reconnect at the same moment? And what does the server do with a client that cannot keep up with the message rate?

Reliable real-time systems answer those questions explicitly: heartbeats detect dead connections, reconnects use backoff with jitter, sequence numbers let clients resume without gaps or duplicates, a backplane fans messages out across servers, and bounded buffers protect the system from slow consumers.

TL;DR

Send heartbeats to detect dead connections and keep intermediaries from closing idle ones.
Reconnect with exponential backoff and full jitter, capped, and reset after a stable connection.
Number messages per stream with sequence numbers; resume with "since seq N", ignore duplicates, and resync on gaps.
Authenticate in the first message or with a short-lived ticket, and handle token expiry with a defined close code.
Fan out through a pub/sub backplane (Redis, NATS, Kafka) so any server can deliver any message.
Cap each connection's send buffer and disconnect or degrade slow consumers; drain connections gradually during deploys.

Quick Example

A browser client that resumes from the last sequence number, reconnects with jittered backoff, and detects gaps:

Server-side fan-out with the Node.js ws library, protecting the server from slow clients:

Core Concepts

Detecting Dead Connections

TCP does not reliably tell either side when the other disappears: a phone that loses signal may leave the server holding a half-open connection for minutes. Heartbeats solve this. Servers can send protocol-level ping frames and expect pong frames; browsers cannot send ping frames from JavaScript, so clients usually send small application-level ping messages. Heartbeats also keep proxies and load balancers from closing idle connections — many default to idle timeouts around 60 seconds, so send heartbeats more often than the shortest timeout in the path.

Reconnecting Well

Ordering, Resumption, and Duplicates

WebSockets deliver messages in order on one connection, but nothing carries across reconnects. Give each stream (a chat room, a document, an account) a monotonic sequence number and let the client resume with the last one it processed. The server replays newer messages from a retained log — an in-memory ring buffer, Redis Streams, or Kafka — or tells the client to resync from a snapshot if the log no longer reaches back that far.

This is at-least-once delivery: after a resume, clients may see messages twice, so they discard sequence numbers they have already applied. Ordering is guaranteed per stream, not globally; if clients combine streams, they need to handle interleaving. See Idempotency.

Authentication and Authorization Refresh

Browsers cannot set custom headers on WebSocket handshakes, so common options are a session cookie (with origin checks against cross-site WebSocket hijacking), a short-lived single-use ticket fetched over HTTPS, or a token in the first message. Avoid long-lived tokens in query strings, which end up in logs. Long-lived connections outlive access tokens: either accept refreshed tokens over the socket before expiry, or close with an application close code (in the 4000–4999 range) that tells the client to fetch a new token and reconnect. Recheck authorization when users lose access to a stream, not only at connect time.

Close Codes

Fan-Out at Scale

With many servers, a message published on server A must reach subscribers connected to servers B and C. A pub/sub backplane — Redis Pub/Sub or Streams, NATS, Kafka, or a managed service — distributes messages to every server, and each server delivers to its local subscribers. See Redis Pub/Sub and Streams.

Slow Consumers and Backpressure

A client on a weak network reads slower than messages arrive. Without limits, the server buffers unboundedly and runs out of memory. Track each connection's buffered bytes and choose a policy: disconnect (the client resumes by sequence number), drop or coalesce updates (send only the latest state for fast-changing values like prices or cursors), or sample. See Quotas, Fairness, and Backpressure.

Deploys and Reconnect Storms

When a server restarts, all its clients reconnect at once — and if every server restarts, the whole fleet does. Drain connections gradually by closing them over a period with 1001 or 1012, rely on client jitter, scale up before rolling deploys, and make sure load balancers spread new connections evenly. Watch connection counts, reconnect rates, resume-versus-resync ratios, and per-connection buffer sizes. See Load Balancing.

Best Practices

Design the Resume Protocol First

Decide on sequence numbers, retention, and snapshot resync before building features, because they are hard to retrofit.

Treat Every Message as Possibly Duplicated

Apply messages idempotently by ID or sequence number, so resumes and retries are harmless.

Bound Everything

Limit buffered bytes per connection, connections per user, message sizes, and subscription counts.

Test Under Network Faults

Simulate drops, latency, sleep and wake, token expiry, and server restarts in automated tests and staging.

Common Mistakes

Reconnecting Immediately in a Tight Loop

Assuming Nothing Was Missed

Unbounded Server Buffers

FAQ

How often should WebSocket heartbeats be sent?

More often than the shortest idle timeout of the proxies and load balancers in the path — commonly every 20–30 seconds — and often enough to detect dead connections within your recovery target.

How do I avoid missing messages after a reconnect?

Assign per-stream sequence numbers, keep a retained log of recent messages, and have clients resume from the last sequence number they processed; fall back to a snapshot when the log does not cover the gap.

Are WebSocket messages delivered in order?

Yes, within a single connection. Across reconnects and multiple servers, ordering requires sequence numbers and a resume protocol.

How should WebSocket connections be authenticated?

With a session cookie and origin checks, a short-lived single-use ticket, or a token sent in the first message — then handle credential expiry and revocation during long-lived connections.

What should a server do with slow clients?

Limit the bytes buffered per connection, then disconnect the client so it resumes later, drop or coalesce updates, or reduce its update rate.