2026-10-15: rolling restart with 10 000 sockets, after #198
The same run as the first 10 000-socket rolling restart, on the branch of
#198: the per-node admission queue for handshakes and replays
(ADR-0077), one database checkout per handshake, and the harness
client reconnecting to a random other node and honouring retryAfterMs.
LOAD_NODES=3 LOAD_SCENARIOS=rolling_restart LOAD_CONVERSATIONS=1000 LOAD_SUBSCRIBERS=10 LOAD_RATE=0.1 \
LOAD_RESTART_CONNECT_RATE=100 LOAD_RESTART_DELAY_S=130 LOAD_WARMUP_S=0 load/run.sh
Same developer machine (Ryzen 5 7600X, 12 vCPUs and 31 GiB for Docker Desktop), nothing else running in Docker.
Before and after
| Metric | Before (a8dd704) | After |
|---|---|---|
| Sockets verified, lost acked messages | 10 000, 0 | 10 000, 0 |
| Client reconnect p50 / p95 / max | 30.0 s / 93.6 s / 175.5 s | 0.9 s / 6.0 s / 7.4 s |
| Reconnects | 12 420 | 15 819 |
| REST → WS p50 / p95 / max during the run | 746 ms / 2.3 s / 12.3 s | 254 ms / 545 ms / 3.6 s |
| REST send retries | 42 307 | 679 |
replay_unavailable answers | 3 449 | 0 |
| DB pool queue wait, max | 1.7 s | 87 ms |
| Nodes ready again after restart | 27–38 s | 21–31 s |
| Messages acked | 21 250 | 28 001 |
The 30 s goal of ADR-0027 is met with room to spare.
Two caveats when comparing:
- Part of the gain is the harness. Before, a dropped client always reconnected to the next node, so every socket of a drained node landed on one survivor, which was restarted next. Now it picks a random other node, the way a load balancer spreads clients. A real deployment behind a load balancer never had the old pile-up.
- More reconnects in the after run is expected: with the old backoff, many clients were still waiting when the run ended.
Results
Commit 145c362be2eb, 2026-10-15T14:07:04Z, profile full. Generated by load/run.sh (see Benchmarks).
| Scenario | Metric | 3 nodes, native |
|---|---|---|
| rolling_restart | node back to ready, max (s) | 31.2 |
| rolling_restart | client reconnect p95 (ms) | 6020.1 |
| rolling_restart | client reconnect max (ms) | 7446 |
| rolling_restart | reconnects | 15819 |
| rolling_restart | sockets verified (zero-loss) | 10000 |
| backbone | activity events observed / expected | 28001 |
| backbone | duplicate events at the subscriber | 0 |
| backbone | send→backbone subscriber p50 (ms) | 14907 |
| backbone | send→backbone subscriber p95 (ms) | 29415 |
| backbone | send→backbone subscriber p99 (ms) | 29615 |
| backbone | caught up after the delivery queues drained (s) | 0.6 |
| rolling_restart | lost acked messages | 0 |
| backbone | lost acked messages | 0 |
Resources
Highest values sampled while the scenario ran (CPU and RSS about every 2.5 s, the rest every second).
| Topology | Scenario | Node | CPU avg % | CPU max % | RSS max (MiB) | DB pool wait max (ms) | Background pool wait max (ms) | Oban queue depth max |
|---|---|---|---|---|---|---|---|---|
| 3 nodes, native | rolling_restart | app | 172.3 | 344.5 | 3501.1 | 86.8 | 111.4 | 2948 |
| 3 nodes, native | rolling_restart | app2 | 154.7 | 306.7 | 2947.1 | 86.8 | 111.4 | 2948 |
| 3 nodes, native | rolling_restart | app3 | 147.4 | 360.6 | 3757.1 | 86.8 | 111.4 | 2948 |
| 3 nodes, native | rolling_restart | db | 125.6 | 253.7 | 776.9 | – | – | – |
Setup
- 3 nodes, native: 12 vCPU and 30.8 GiB visible to Docker (MINGW64_NT-10.0-26200, local, kernel 6.6.87.2-microsoft-standard-WSL2, Docker 29.8.2); per-node limits CPU unlimited, memory unlimited; Postgres 17.11, 185.3 MiB after the run; pool size 20 + 10 background; mock provider latency 20+10 ms, error rate 0.
rolling_restart: mode=restart, conversations=1000, subscribers_per_conversation=10, sockets=10000, connect_rate=100, rate_per_conversation=0.1, duration_s=280, storm_at_s=null, storm_every_s=nullbackbone: backbone=native, subscriber=platform webhook (activity.created)
Checks
- ✅ 3 nodes, native: rolling_restart: k6 ran and correctness thresholds passed (k6 exit 0)
- ✅ 3 nodes, native: rolling_restart: messages sent and received live (28001 acked, 276753 latency samples)
- ✅ 3 nodes, native: backbone (native): every activity observed by the subscriber exactly once (28001/28001 observed, 28001 webhook events (0 duplicates), 0 HTTP redeliveries)
- ✅ 3 nodes, native: rolling_restart: zero lost acked messages (lost 0, never acked 0)
- ✅ 3 nodes, native: fan-out channel: every activity delivered to the outbound webhook (28001/28001 delivered, 0 jobs still open)