How NATS Lame Duck Mode Actually Works

Every production NATS deployment eventually needs a rolling restart. Lame duck mode is how NATS coordinates graceful server shutdown without dropping all connections simultaneously. The mechanism is straightforward once you see the moving parts, but most operators only encounter it during their first maintenance window – usually under pressure.

The Protocol

When a NATS server enters lame duck mode (via SIGUSR2, nats-server --signal ldm, or programmatic LameDuckShutdown() call), it executes a multi-phase shutdown:¹

1. Stops accepting new connections (closes the listener)
2. Transfers Raft leadership roles (if JetStream enabled)
3. Shuts down JetStream
4. Shuts down Raft nodes
5. Waits for the accept loops to fully drain
6. Sends an INFO protocol message with ldm: true (and an updated connect_urls list that no longer advertises this server) to every connected client²
7. Waits through a grace period (default 10 seconds)
8. Gradually disconnects clients over the remaining duration

The client side responds automatically:

1. Detects the ldm: true flag in the INFO message
2. Fires the LameDuckModeHandler callback if configured
3. Removes the draining server from its connection pool – but only because the server’s INFO update no longer advertises itself in connect_urls. The ldm: true flag alone does not trigger pool removal; the standard connect_urls update path does
4. When disconnected, reconnects to one of the alternative servers provided in the INFO message
5. Restores all subscriptions on the new connection

The result: clients migrate to healthy servers without application-level intervention.

Timing

The defaults balance two competing concerns. Disconnecting clients too fast causes a reconnection storm on the remaining servers. Disconnecting too slowly delays the maintenance operation.

DEFAULT_LAME_DUCK_DURATION    = 2 * time.Minute  // spread disconnections
DEFAULT_LAME_DUCK_GRACE_PERIOD = 10 * time.Second // initial breathing room

These constants are configurable via the server config.³

The server calculates a sleep interval between disconnections:

interval = (duration - grace_period) / number_of_clients

For 1,000 clients with default settings, that’s roughly 110ms between each disconnection. The server randomizes each sleep to between 50% and 100% of the calculated interval to prevent synchronized reconnection waves.⁴ number_of_clients is snapshotted once after the accept loops drain, before any client receives the LDM INFO; clients that disconnect themselves during the grace period reduce the actual count but do not shorten the sleep interval.

For large deployments (10k+ connections), increase the duration:

lame_duck_duration: 5m
lame_duck_grace_period: 30s

The Handler

Most stateless services don’t need a handler – the automatic reconnection is sufficient. For stateful services, the handler provides a window to prepare:

nc, err := nats.Connect(servers,
    nats.LameDuckModeHandler(func(nc *nats.Conn) {
        log.Info("server draining, preparing for reconnection")
        completeInFlightWork()
        pauseNewRequests()
    }),
)

The handler runs asynchronously and should not block for long. Its purpose is to signal your application to wrap up current work before the disconnection arrives.

Rolling Restart Sequence

For a 3-node cluster:

# Node 1 -- preferred: signal via the CLI (no PID required, works cross-platform)
nats-server --signal ldm

# Or via Unix signal (Linux: pidof; macOS: pgrep -x)
kill -USR2 $(pgrep -x nats-server)

# Wait for connections to drain, then restart with new version
systemctl restart nats-server

# Verify node 1 is healthy before proceeding
nats server report jetstream

# Repeat for nodes 2 and 3, one at a time

Wait for each node to fully rejoin the cluster and restore its Raft group memberships before proceeding to the next. The nats server report jetstream output confirms when all stream replicas are healthy. If a node fails to rejoin – particularly after a name change – see Recovering a JetStream Cluster After Quorum Loss for the recovery procedure.

Health Checks

During lame duck mode, the server’s /healthz endpoint returns 500 Internal Server Error (the listener is closed, so the readiness check fails).⁵ If you’re running behind a load balancer or using Kubernetes readiness probes, this automatically stops new traffic from being routed to the draining server.

In Kubernetes:

readinessProbe:
  httpGet:
    path: /healthz
    port: 8222

The readiness probe will fail as soon as lame duck mode begins, and Kubernetes will stop sending new connections to that pod before the grace period even expires.

Common Mistakes

Using SIGKILL during lame duck. The whole point is graceful shutdown. Sending SIGKILL after SIGUSR2 interrupts the client migration mid-flight, causing exactly the connection storm lame duck mode is designed to prevent.

Not accounting for in-flight work. The grace period delays the start of client disconnections, but does not guarantee each client time to finish work. Use the LameDuckModeHandler callback to drain in-flight requests, and ensure lame_duck_duration is long enough for all clients to be disconnected gracefully. Note that lame_duck_duration is not a strict ceiling on total drain time – Raft leader transfer (up to ~1 second) and JetStream shutdown run before the duration timer starts, so the actual elapsed time from LDM entry to final disconnect exceeds the configured duration.

Skipping lame duck for “quick” restarts. Even a fast restart drops all connections simultaneously, causing a thundering herd on the remaining servers. Lame duck mode takes 2 minutes by default.