2026-10-15, a8dd704: rolling restart with 10 000 sockets
This run measures the exit criterion of #58 that was tracked in
#190: a rolling restart under 10 000 connections loses no
acknowledged message. The application code is main at d23f8ce; a8dd704 adds the gradual connect
(LOAD_RESTART_CONNECT_RATE) and the restart delay (LOAD_RESTART_DELAY_S) to the harness.
LOAD_NODES=3 LOAD_SCENARIOS=rolling_restart LOAD_CONVERSATIONS=1000 LOAD_SUBSCRIBERS=10 LOAD_RATE=0.1 \
LOAD_RESTART_CONNECT_RATE=100 LOAD_RESTART_DELAY_S=130 LOAD_WARMUP_S=0 load/run.sh
1 000 conversations with 10 sockets each connect at 100 sockets/s, which takes 100 s. Every conversation then gets a
REST message every 10 s (100 messages/s in total, 1 000 WebSocket frames/s). After 130 s, run.sh restarts the three
nodes one after the other with SIGTERM: readiness flips, sockets drain with close 1012, and the node comes back. Each
of the 10 000 sockets resumes from its last seq and must end with every seq of its conversation exactly once.
It ran on the same developer machine as the first run: Windows 11 with Docker Desktop (WSL 2), 12 vCPUs and 31 GiB, shared by the three nodes, Postgres, k6 and the mock provider. Nothing else was running in Docker; other desktop applications were open.
Findings
- No acknowledged message was lost. All 10 000 sockets were verified: 21 250 messages acked, 0 lost, 0 duplicate
seq, 0 never acked. Every message also reached the outbound webhook and the backbone subscriber exactly once. - Reconnects are far slower than the 30 s target of ADR-0027.
The time from close to
welcomewas p50 30.0 s, p95 93.6 s and max 175.5 s. When a node drains, its ~3 300 sockets move to the two other nodes. Every handshake and every watermark replay reads the database, so the 20-connection pools saturated (1.7 s queue wait at the peak). Requests were answered with retryable errors (3 449replay_unavailable, 42 307 REST send retries), and client backoff stretched the tail. This is tracked in #198. - Nodes were ready again 27–38 s after their restart, and REST → WebSocket latency during the run was p50 746 ms, p95 2.3 s.
- One machine ran the whole cluster, Postgres and the load generator, so absolute numbers are pessimistic. The zero-loss result does not depend on that.
Results
Commit a8dd704fe1d2, 2026-10-15T02:17:28Z, profile full. Generated by load/run.sh (see Benchmarks).
| Scenario | Metric | 3 nodes, native |
|---|---|---|
| rolling_restart | node back to ready, max (s) | 38.2 |
| rolling_restart | client reconnect p95 (ms) | 93572.9 |
| rolling_restart | client reconnect max (ms) | 175524 |
| rolling_restart | reconnects | 12420 |
| rolling_restart | sockets verified (zero-loss) | 10000 |
| backbone | activity events observed / expected | 21250 |
| backbone | duplicate events at the subscriber | 0 |
| backbone | send→backbone subscriber p50 (ms) | 71883 |
| backbone | send→backbone subscriber p95 (ms) | 102943 |
| backbone | send→backbone subscriber p99 (ms) | 105621 |
| backbone | caught up after the delivery queues drained (s) | 1.5 |
| rolling_restart | lost acked messages | 0 |
| rolling_restart | holes recovered with sync / replay_unavailable errors | 621 / 3449 |
| backbone | lost acked messages | 0 |
Resources
Highest values sampled while the scenario ran (CPU and RSS about every 2.5 s, the rest every second).
| Topology | Scenario | Node | CPU avg % | CPU max % | RSS max (MiB) | DB pool wait max (ms) | Background pool wait max (ms) | Oban queue depth max |
|---|---|---|---|---|---|---|---|---|
| 3 nodes, native | rolling_restart | app | 146.7 | 304.4 | 1886.2 | 1713.8 | 1025 | 10917 |
| 3 nodes, native | rolling_restart | app2 | 144.7 | 256.5 | 1863.7 | 1713.8 | 1025 | 10917 |
| 3 nodes, native | rolling_restart | app3 | 137.1 | 239.1 | 2032.6 | 1713.8 | 1025 | 10917 |
| 3 nodes, native | rolling_restart | db | 125.7 | 318.2 | 737.3 | – | – | – |
Setup
- 3 nodes, native: 12 vCPU and 30.8 GiB visible to Docker (MINGW64_NT-10.0-26200, local, kernel 6.6.87.2-microsoft-standard-WSL2, Docker 29.8.2); per-node limits CPU unlimited, memory unlimited; Postgres 17.11, 151 MiB after the run; pool size 20 + 10 background; mock provider latency 20+10 ms, error rate 0.
rolling_restart: mode=restart, conversations=1000, subscribers_per_conversation=10, sockets=10000, connect_rate=100, rate_per_conversation=0.1, duration_s=280, storm_at_s=null, storm_every_s=nullbackbone: backbone=native, subscriber=platform webhook (activity.created)
Checks
- ✅ 3 nodes, native: rolling_restart: k6 ran and correctness thresholds passed (k6 exit 0)
- ✅ 3 nodes, native: rolling_restart: messages sent and received live (21250 acked, 186817 latency samples)
- ✅ 3 nodes, native: backbone (native): every activity observed by the subscriber exactly once (21250/21250 observed, 21250 webhook events (0 duplicates), 0 HTTP redeliveries)
- ✅ 3 nodes, native: rolling_restart: zero lost acked messages (lost 0, never acked 0)
- ✅ 3 nodes, native: fan-out channel: every activity delivered to the outbound webhook (21250/21250 delivered, 0 jobs still open)