2026-10-12, 612ec15: first run, 1 and 3 nodes
The first full run of the load harness, full profile, on the native backbone. The application code is
main at 92e21a1; 612ec15 only adds the harness. This is the pre-scale baseline that
#34 asks to compare against after
#35. The Kafka column comes with
#66.
It ran on a developer machine: Windows 11 with Docker Desktop (WSL 2), 12 vCPUs and 31 GiB visible to Docker. Every node, Postgres, k6 and the mock provider shared these, with no per-node limits. In the 3-node topology the three nodes also share one host and one Postgres (3 × 20 pooled connections). That topology therefore measures clustering overhead and failover, not horizontal scale. These numbers are not comparable with the nightly CI runs.
Findings
- No acknowledged message was lost in any scenario on either topology. This held for 1 000 sockets under load, reconnect storms of all 1 000 sockets, graceful restarts of every node, and a SIGKILL. It also covers every webhook delivery (8 634 and 6 789 burst deliveries, 23 820 and 27 382 fan-out deliveries).
- REST → WebSocket p95 misses the 250 ms target: 433 ms on 1 node at 94 msgs/s with 1 000 sockets. The REST POST itself takes as long (432 ms p95), so the write path is the bottleneck, not the WebSocket fan-out.
- The DB pool saturates under bursts. At 500 inbound webhooks/s, one node with
POOL_SIZE=20accepted about 96/s. The rest got HTTP 500: Ecto dropped the queued connection checkouts after about 100 ms. Converger should answer 503 or 429 withRetry-Afterand flip readiness instead, which is the subject of #35. WebSocket handshakes also read the database: in an earlier attempt at 500 connects/s, only 2 579 of 10 000 sockets got in (the harness now connects at 100/s and clients back off). - Outbound delivery throughput is low: about 85 deliveries/s with a mock provider that answers in 20–30 ms and 20 delivery workers, where about 700/s would be expected. After the burst, the p50 delivery took 42 s.
- Idle sockets cost about 130–150 KiB each (RSS growth for 10 000 sockets: 1.9 GB on one node). That is more than the "tens of KiB" in the capacity guidance.
- Nodes come back ready 20–30 s after a restart or kill (42 s at most on 3 nodes). During a 3-node rolling restart, the slowest client needed 51 s to reconnect: its backoff kept landing on nodes that were restarting or busy.
Results
Measured commit 612ec15a4a6d, 2026-10-12T00:46:37Z, profile full. Generated by load/run.sh (see Benchmarks).
| Scenario | Metric | 1 node, native | 3 nodes, native |
|---|---|---|---|
| e2e | REST→WS p50 (ms) | 80 | 502 |
| e2e | REST→WS p95 (ms) | 433 | 1149 |
| e2e | REST→WS p99 (ms) | 548 | 2053 |
| e2e | REST POST p95 (ms) | 431.6 | 1147.7 |
| e2e | sustained msgs/s (acked) | 93.7 | 91.3 |
| e2e | WS frames/s delivered | 937.2 | 913.2 |
| e2e | WS connections | 1000 | 1000 |
| webhook_burst | inbound accepted/s | 95.9 | 75.4 |
| webhook_burst | inbound POST p95 (ms) | 3213.2 | 5098.9 |
| webhook_burst | inbound→provider p50 (ms) | 42454 | 28474 |
| webhook_burst | inbound→provider p95 (ms) | 56310 | 56937 |
| webhook_burst | inbound→provider p99 (ms) | 89236 | 61747 |
| webhook_burst | deliveries/s | 84.5 | 87.8 |
| webhook_burst | queue drained after burst (s) | 54.3 | 31.6 |
| idle | idle sockets connected | 10000 | 10000 |
| idle | connect p95 (ms) | 265 | 293 |
| idle | memory per idle socket (KiB) | 149 | 127.5 |
| reconnect_storm | reconnect p95 (ms) | 1834 | 6883 |
| reconnect_storm | reconnect max (ms) | 1998 | 7408 |
| reconnect_storm | REST→WS p95 during storms (ms) | 1085 | 1783 |
| rolling_restart | node back to ready, max (s) | 26.5 | 42 |
| rolling_restart | client reconnect max (ms) | 17216 | 50853 |
| node_kill | killed node back to ready (s) | 20.3 | 28.5 |
| node_kill | client reconnect max (ms) | 14717 | 15843 |
| e2e | lost acked messages | 0 | 0 |
| webhook_burst | lost acked messages | 0 | 0 |
| reconnect_storm | lost acked messages | 0 | 0 |
| rolling_restart | lost acked messages | 0 | 0 |
| node_kill | lost acked messages | 0 | 0 |
Resources
Highest values sampled while the scenario ran (CPU and RSS about every 2.5 s, the rest every second).
| Topology | Scenario | Node | CPU avg % | CPU max % | RSS max (MiB) | DB pool wait max (ms) | Oban queue depth max |
|---|---|---|---|---|---|---|---|
| 1 node, native | e2e | app | 251.3 | 435.2 | 678 | 3.9 | 112 |
| 1 node, native | e2e | db | 139.4 | 270.3 | 199.4 | – | – |
| 1 node, native | webhook_burst | app | 263.5 | 430.7 | 500.4 | 476.3 | 10235 |
| 1 node, native | webhook_burst | db | 208 | 422.8 | 234.7 | – | – |
| 1 node, native | idle | app | 67.4 | 282.7 | 1883.1 | 0.5 | 0 |
| 1 node, native | idle | db | 13.9 | 95.8 | 236.4 | – | – |
| 1 node, native | reconnect_storm | app | 322.7 | 562.6 | 1559.6 | 266 | 222 |
| 1 node, native | reconnect_storm | db | 152 | 266.1 | 250.4 | – | – |
| 1 node, native | rolling_restart | app | 279.1 | 497.8 | 1540.1 | 78.8 | 231 |
| 1 node, native | rolling_restart | db | 147.4 | 250.5 | 274.3 | – | – |
| 1 node, native | node_kill | app | 211.7 | 437.7 | 1305.6 | 373.4 | 198 |
| 1 node, native | node_kill | db | 103.6 | 249.3 | 257.9 | – | – |
| 3 nodes, native | e2e | app | 119 | 186.4 | 397.5 | 271.5 | 131 |
| 3 nodes, native | e2e | app2 | 116.2 | 177.7 | 434.4 | 271.5 | 131 |
| 3 nodes, native | e2e | app3 | 119.9 | 181.4 | 483.7 | 271.5 | 131 |
| 3 nodes, native | e2e | db | 139.6 | 227.9 | 427.7 | – | – |
| 3 nodes, native | webhook_burst | app | 114.5 | 202.5 | 374.9 | 1671.1 | 7938 |
| 3 nodes, native | webhook_burst | app2 | 125.1 | 293.4 | 467.5 | 1671.1 | 7938 |
| 3 nodes, native | webhook_burst | app3 | 132 | 280.8 | 385.6 | 1671.1 | 7938 |
| 3 nodes, native | webhook_burst | db | 279.3 | 574.3 | 482 | – | – |
| 3 nodes, native | idle | app | 32 | 92.7 | 820.2 | 61.8 | 0 |
| 3 nodes, native | idle | app2 | 32 | 82.8 | 763.5 | 61.8 | 0 |
| 3 nodes, native | idle | app3 | 33.7 | 86.5 | 814.7 | 61.8 | 0 |
| 3 nodes, native | idle | db | 22 | 130.3 | 488.4 | – | – |
| 3 nodes, native | reconnect_storm | app | 103.1 | 196.8 | 599.3 | 578.5 | 282 |
| 3 nodes, native | reconnect_storm | app2 | 101 | 207.8 | 622.1 | 578.5 | 282 |
| 3 nodes, native | reconnect_storm | app3 | 112.7 | 221.6 | 751.5 | 578.5 | 282 |
| 3 nodes, native | reconnect_storm | db | 118.1 | 208.4 | 500.9 | – | – |
| 3 nodes, native | rolling_restart | app | 120.8 | 450.7 | 640.9 | 1130.3 | 1124 |
| 3 nodes, native | rolling_restart | app2 | 140.5 | 516.2 | 790 | 1130.3 | 1124 |
| 3 nodes, native | rolling_restart | app3 | 126.3 | 501.6 | 747.1 | 1130.3 | 1124 |
| 3 nodes, native | rolling_restart | db | 162.7 | 621.8 | 509.1 | – | – |
| 3 nodes, native | node_kill | app | 79.6 | 191.9 | 767.3 | 500.8 | 528 |
| 3 nodes, native | node_kill | app2 | 145.6 | 281.7 | 792.5 | 500.8 | 528 |
| 3 nodes, native | node_kill | app3 | 144.3 | 438.5 | 691.1 | 500.8 | 528 |
| 3 nodes, native | node_kill | db | 121.3 | 322.8 | 505.8 | – | – |
Setup
- 1 node, native: 12 vCPU and 30.8 GiB visible to Docker (MINGW64_NT-10.0-26200, local, kernel 6.6.87.2-microsoft-standard-WSL2, Docker 29.2.1); per-node limits CPU unlimited, memory unlimited; Postgres 17.10, 79.4 MiB after the run; pool size 20; mock provider latency 20+10 ms, error rate 0.
e2e: conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=60webhook_burst: rate=500, duration_s=30, participants=200, targets=3idle: sockets=10000, sockets_per_vu=200, connect_rate=100, ramp_s=100, hold_s=60reconnect_storm: mode=storm, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=60, storm_at_s=15, storm_every_s=15rolling_restart: mode=restart, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=70, storm_at_s=null, storm_every_s=nullnode_kill: mode=restart, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=70, storm_at_s=null, storm_every_s=null
- 3 nodes, native: 12 vCPU and 30.8 GiB visible to Docker (MINGW64_NT-10.0-26200, local, kernel 6.6.87.2-microsoft-standard-WSL2, Docker 29.2.1); per-node limits CPU unlimited, memory unlimited; Postgres 17.10, 83.6 MiB after the run; pool size 20; mock provider latency 20+10 ms, error rate 0.
e2e: conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=60webhook_burst: rate=500, duration_s=30, participants=200, targets=3idle: sockets=10000, sockets_per_vu=200, connect_rate=100, ramp_s=100, hold_s=60reconnect_storm: mode=storm, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=60, storm_at_s=15, storm_every_s=15rolling_restart: mode=restart, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=170, storm_at_s=null, storm_every_s=nullnode_kill: mode=restart, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=70, storm_at_s=null, storm_every_s=null
Checks
- ✅ 1 node, native: e2e: k6 ran and correctness thresholds passed (k6 exit 99)
- ⚠️ 1 node, native: e2e: performance targets (missed: e2e_latency_ms: p(95)<250)
- ✅ 1 node, native: e2e: messages sent and received live (5623 acked, 56230 latency samples)
- ✅ 1 node, native: webhook_burst: k6 ran and correctness thresholds passed (k6 exit 99)
- ⚠️ 1 node, native: webhook_burst: performance targets (missed: inbound_rejected: count<1, dropped_iterations: count<1, http_req_duration{name:POST /inbound}: p(95)<500)
- ✅ 1 node, native: webhook_burst: every accepted message stored and delivered to every target (accepted 2878, stored 2878, delivered 8634/8634)
- ✅ 1 node, native: idle: k6 ran and correctness thresholds passed (k6 exit 0)
- ✅ 1 node, native: reconnect_storm: k6 ran and correctness thresholds passed (k6 exit 0)
- ✅ 1 node, native: reconnect_storm: messages sent and received live (5380 acked, 52711 latency samples)
- ✅ 1 node, native: rolling_restart: k6 ran and correctness thresholds passed (k6 exit 0)
- ✅ 1 node, native: rolling_restart: messages sent and received live (5987 acked, 55852 latency samples)
- ✅ 1 node, native: node_kill: k6 ran and correctness thresholds passed (k6 exit 0)
- ✅ 1 node, native: node_kill: messages sent and received live (5331 acked, 52016 latency samples)
- ✅ 1 node, native: e2e: zero lost acked messages (lost 0, never acked 0)
- ✅ 1 node, native: reconnect_storm: zero lost acked messages (lost 0, never acked 0)
- ✅ 1 node, native: rolling_restart: zero lost acked messages (lost 0, never acked 0)
- ✅ 1 node, native: node_kill: zero lost acked messages (lost 0, never acked 0)
- ✅ 1 node, native: fan-out channel: every activity delivered to the outbound webhook (23820/23820 delivered, 0 jobs still open)
- ✅ 3 nodes, native: e2e: k6 ran and correctness thresholds passed (k6 exit 99)
- ⚠️ 3 nodes, native: e2e: performance targets (missed: http_req_duration{name:POST /activities}: p(95)<1000, e2e_latency_ms: p(95)<250)
- ✅ 3 nodes, native: e2e: messages sent and received live (5479 acked, 54790 latency samples)
- ✅ 3 nodes, native: webhook_burst: k6 ran and correctness thresholds passed (k6 exit 99)
- ⚠️ 3 nodes, native: webhook_burst: performance targets (missed: inbound_rejected: count<1, dropped_iterations: count<1, http_req_duration{name:POST /inbound}: p(95)<500)
- ✅ 3 nodes, native: webhook_burst: every accepted message stored and delivered to every target (accepted 2263, stored 2263, delivered 6789/6789)
- ✅ 3 nodes, native: idle: k6 ran and correctness thresholds passed (k6 exit 0)
- ✅ 3 nodes, native: reconnect_storm: k6 ran and correctness thresholds passed (k6 exit 0)
- ✅ 3 nodes, native: reconnect_storm: messages sent and received live (5002 acked, 46969 latency samples)
- ✅ 3 nodes, native: rolling_restart: k6 ran and correctness thresholds passed (k6 exit 0)
- ✅ 3 nodes, native: rolling_restart: messages sent and received live (10292 acked, 94478 latency samples)
- ✅ 3 nodes, native: node_kill: k6 ran and correctness thresholds passed (k6 exit 0)
- ✅ 3 nodes, native: node_kill: messages sent and received live (5109 acked, 50328 latency samples)
- ✅ 3 nodes, native: e2e: zero lost acked messages (lost 0, never acked 0)
- ✅ 3 nodes, native: reconnect_storm: zero lost acked messages (lost 0, never acked 0)
- ✅ 3 nodes, native: rolling_restart: zero lost acked messages (lost 0, never acked 0)
- ✅ 3 nodes, native: node_kill: zero lost acked messages (lost 0, never acked 0)
- ✅ 3 nodes, native: fan-out channel: every activity delivered to the outbound webhook (27382/27382 delivered, 0 jobs still open)