Backbone matrix
How the event backbone (ADR-0070) changes the numbers of the
load harness: every scenario on the Native backbone (Phoenix.PubSub + Oban on Postgres) and on Kafka,
with one and three nodes. The backbone rows are the backbone zero-loss check: every activity of the run is
published to the backbone, and a platform webhook subscriber must observe each one exactly once.
Measured with load/matrix.sh on DESKTOP-Q8TVMOL (2026-10-14).
Commit e1c2862cfe81, 2026-10-14T22:11:16Z, profile full. Generated by load/run.sh (see Benchmarks).
| Scenario | Metric | 1 node, native | 3 nodes, native | 1 node, kafka | 3 nodes, kafka |
|---|---|---|---|---|---|
| e2e | REST→WS p50 (ms) | 53 | 208 | 41 | 77 |
| e2e | REST→WS p95 (ms) | 114 | 346 | 78 | 144 |
| e2e | REST→WS p99 (ms) | 175 | 394 | 117 | 178 |
| e2e | REST POST p95 (ms) | 113.3 | 345.7 | 77.8 | 143 |
| e2e | sustained msgs/s (acked) | 100 | 100 | 100 | 100 |
| e2e | WS frames/s delivered | 1000 | 1000 | 1000 | 1000 |
| e2e | WS connections | 1000 | 1000 | 1000 | 1000 |
| webhook_burst | inbound accepted/s | 142.8 | 107.4 | 128 | 88 |
| webhook_burst | inbound POST p95 (ms) | 1469.5 | 1803.1 | 1546.1 | 1662.5 |
| webhook_burst | inbound→provider p50 (ms) | 37272 | 25373 | 27623 | 9826 |
| webhook_burst | inbound→provider p95 (ms) | 46747 | 29070 | 36564 | 12653 |
| webhook_burst | inbound→provider p99 (ms) | 48297 | 29716 | 36985 | 13389 |
| webhook_burst | deliveries/s | 161.6 | 177.1 | 164.7 | 183.1 |
| webhook_burst | queue drained after burst (s) | 46.5 | 20.9 | 36.6 | 9.2 |
| idle | idle sockets connected | 10000 | 10000 | 10000 | 10000 |
| idle | connect p95 (ms) | 6 | 7 | 6 | 7 |
| idle | memory per idle socket (KiB) | 108.7 | 105.8 | 109.9 | 113.4 |
| reconnect_storm | reconnect p95 (ms) | 887 | 1061 | 1545.1 | 1318 |
| reconnect_storm | reconnect max (ms) | 1046 | 1180 | 1778 | 1499 |
| reconnect_storm | REST→WS p95 during storms (ms) | 233 | 461 | 355 | 509 |
| rolling_restart | node back to ready, max (s) | 10.3 | 13.7 | 40.6 | 43.1 |
| rolling_restart | client reconnect max (ms) | 10154 | 2023 | 49131 | 2685 |
| node_kill | killed node back to ready (s) | 5.1 | 7.9 | 5.7 | 10 |
| node_kill | client reconnect max (ms) | 10080 | 900 | 10337 | 924 |
| backbone | activity events observed / expected | 31149 | 40629 | 27667 | 39973 |
| backbone | duplicate events at the subscriber | 0 | 0 | 0 | 0 |
| backbone | send→backbone subscriber p50 (ms) | 243 | 2340 | 357 | 3823 |
| backbone | send→backbone subscriber p95 (ms) | 12484 | 11493 | 28162 | 39689 |
| backbone | send→backbone subscriber p99 (ms) | 13609 | 16783 | 30227 | 46839 |
| backbone | caught up after the delivery queues drained (s) | 0.3 | 0.4 | 0.5 | 2.2 |
| e2e | lost acked messages | 0 | 0 | 0 | 0 |
| webhook_burst | lost acked messages | 0 | 0 | 0 | 0 |
| reconnect_storm | lost acked messages | 0 | 0 | 0 | 0 |
| rolling_restart | lost acked messages | 0 | 0 | 0 | 0 |
| node_kill | lost acked messages | 0 | 0 | 0 | 0 |
| node_kill | holes recovered with sync / replay_unavailable errors | 54 / 58 | 325 / 390 | 25 / 25 | 420 / 464 |
| backbone | lost acked messages | 0 | 0 | 0 | 0 |
Resources
Highest values sampled while the scenario ran (CPU and RSS about every 2.5 s, the rest every second).
| Topology | Scenario | Node | CPU avg % | CPU max % | RSS max (MiB) | DB pool wait max (ms) | Background pool wait max (ms) | Oban queue depth max |
|---|---|---|---|---|---|---|---|---|
| 1 node, native | e2e | app | 391.7 | 573.7 | 1428.5 | 85 | 15.4 | 136 |
| 1 node, native | e2e | db | 151.8 | 246 | 304.7 | – | – | – |
| 1 node, native | webhook_burst | app | 359 | 581.6 | 650.2 | 449.9 | 16.1 | 15866 |
| 1 node, native | webhook_burst | db | 256.8 | 478.3 | 367.8 | – | – | – |
| 1 node, native | idle | app | 59.6 | 435.3 | 1627.1 | 0 | 340 | 0 |
| 1 node, native | idle | db | 14.4 | 41.8 | 375 | – | – | – |
| 1 node, native | reconnect_storm | app | 377.4 | 595.6 | 1626.1 | 98.3 | 10.1 | 189 |
| 1 node, native | reconnect_storm | db | 143.1 | 265.8 | 394.7 | – | – | – |
| 1 node, native | rolling_restart | app | 377.3 | 589 | 1610.8 | 6.8 | 9.1 | 297 |
| 1 node, native | rolling_restart | db | 121.6 | 213.8 | 397.5 | – | – | – |
| 1 node, native | node_kill | app | 346.4 | 548.7 | 1380.4 | 168.4 | 27.4 | 671 |
| 1 node, native | node_kill | db | 116.8 | 372.6 | 463.1 | – | – | – |
| 3 nodes, native | e2e | app | 160.9 | 216.3 | 543.9 | 6.7 | 27.1 | 377 |
| 3 nodes, native | e2e | app2 | 150.3 | 205.5 | 623.6 | 6.7 | 27.1 | 377 |
| 3 nodes, native | e2e | app3 | 163.8 | 197.9 | 581.7 | 6.7 | 27.1 | 377 |
| 3 nodes, native | e2e | db | 169 | 241.2 | 617.6 | – | – | – |
| 3 nodes, native | webhook_burst | app | 183.5 | 216.6 | 456.9 | 417.1 | 55.9 | 11523 |
| 3 nodes, native | webhook_burst | app2 | 183.6 | 235.4 | 461.7 | 417.1 | 55.9 | 11523 |
| 3 nodes, native | webhook_burst | app3 | 187.7 | 241.5 | 468 | 417.1 | 55.9 | 11523 |
| 3 nodes, native | webhook_burst | db | 306.2 | 422.7 | 707.4 | – | – | – |
| 3 nodes, native | idle | app | 26 | 71.4 | 781.4 | 0.1 | 223.3 | 0 |
| 3 nodes, native | idle | app2 | 26.7 | 68.8 | 797.9 | 0.1 | 223.3 | 0 |
| 3 nodes, native | idle | app3 | 25.1 | 72.6 | 820.6 | 0.1 | 223.3 | 0 |
| 3 nodes, native | idle | db | 15.5 | 45.8 | 718.2 | – | – | – |
| 3 nodes, native | reconnect_storm | app | 160.8 | 232.3 | 809.3 | 139.3 | 35.8 | 451 |
| 3 nodes, native | reconnect_storm | app2 | 154.5 | 212.8 | 827.5 | 139.3 | 35.8 | 451 |
| 3 nodes, native | reconnect_storm | app3 | 160.4 | 246.5 | 904.9 | 139.3 | 35.8 | 451 |
| 3 nodes, native | reconnect_storm | db | 155 | 291 | 769.2 | – | – | – |
| 3 nodes, native | rolling_restart | app | 177.9 | 347.2 | 1123.3 | 168.1 | 54.3 | 1061 |
| 3 nodes, native | rolling_restart | app2 | 153.1 | 286.1 | 774.1 | 168.1 | 54.3 | 1061 |
| 3 nodes, native | rolling_restart | app3 | 153.8 | 305.2 | 1216.5 | 168.1 | 54.3 | 1061 |
| 3 nodes, native | rolling_restart | db | 164 | 302.8 | 767.7 | – | – | – |
| 3 nodes, native | node_kill | app | 131.1 | 219.4 | 754 | 206.8 | 19.7 | 123 |
| 3 nodes, native | node_kill | app2 | 176.1 | 293.1 | 734.2 | 206.8 | 19.7 | 123 |
| 3 nodes, native | node_kill | app3 | 152.8 | 239.3 | 666.4 | 206.8 | 19.7 | 123 |
| 3 nodes, native | node_kill | db | 168.2 | 373.1 | 851.4 | – | – | – |
| 1 node, kafka | e2e | app | 438.7 | 544.3 | 883.4 | 5 | 0.7 | 20 |
| 1 node, kafka | e2e | db | 154.3 | 231.9 | 292.8 | – | – | – |
| 1 node, kafka | e2e | kafka | 57.1 | 214.7 | 899.4 | – | – | – |
| 1 node, kafka | webhook_burst | app | 365.6 | 477.3 | 622.9 | 324.1 | 2.2 | 10266 |
| 1 node, kafka | webhook_burst | db | 308 | 496.9 | 354 | – | – | – |
| 1 node, kafka | webhook_burst | kafka | 45.1 | 177.9 | 1070.1 | – | – | – |
| 1 node, kafka | idle | app | 69.2 | 444.8 | 1664 | 0.1 | 118 | 0 |
| 1 node, kafka | idle | db | 14.3 | 54 | 359.1 | – | – | – |
| 1 node, kafka | idle | kafka | 31.5 | 170.3 | 1076.2 | – | – | – |
| 1 node, kafka | reconnect_storm | app | 413.9 | 572.4 | 1700.9 | 106.1 | 2.7 | 74 |
| 1 node, kafka | reconnect_storm | db | 171.7 | 318.1 | 390.6 | – | – | – |
| 1 node, kafka | reconnect_storm | kafka | 46.7 | 181 | 1085.4 | – | – | – |
| 1 node, kafka | rolling_restart | app | 239.6 | 536.4 | 1735.7 | 136.1 | 1.4 | 211 |
| 1 node, kafka | rolling_restart | db | 81.6 | 237.3 | 424.1 | – | – | – |
| 1 node, kafka | rolling_restart | kafka | 35.4 | 193.1 | 1093.6 | – | – | – |
| 1 node, kafka | node_kill | app | 374.3 | 548 | 1379.3 | 95 | 3.1 | 97 |
| 1 node, kafka | node_kill | db | 130.1 | 331.9 | 438 | – | – | – |
| 1 node, kafka | node_kill | kafka | 45.9 | 197.9 | 1109 | – | – | – |
| 3 nodes, kafka | e2e | app | 186.1 | 222.3 | 601.4 | 16.7 | 1.1 | 17 |
| 3 nodes, kafka | e2e | app2 | 184.2 | 221.5 | 522.1 | 16.7 | 1.1 | 17 |
| 3 nodes, kafka | e2e | app3 | 178.9 | 221.6 | 491.2 | 16.7 | 1.1 | 17 |
| 3 nodes, kafka | e2e | db | 193.5 | 280.1 | 618 | – | – | – |
| 3 nodes, kafka | e2e | kafka | 58.2 | 179.6 | 811.2 | – | – | – |
| 3 nodes, kafka | webhook_burst | app | 194.7 | 220.6 | 484.4 | 375.5 | 11.2 | 4000 |
| 3 nodes, kafka | webhook_burst | app2 | 192.3 | 218.2 | 469.7 | 375.5 | 11.2 | 4000 |
| 3 nodes, kafka | webhook_burst | app3 | 199.3 | 240.9 | 474.4 | 375.5 | 11.2 | 4000 |
| 3 nodes, kafka | webhook_burst | db | 365.4 | 439.7 | 705.1 | – | – | – |
| 3 nodes, kafka | webhook_burst | kafka | 46.4 | 140.4 | 968.3 | – | – | – |
| 3 nodes, kafka | idle | app | 29.9 | 193.4 | 828.1 | 0.1 | 9.7 | 11 |
| 3 nodes, kafka | idle | app2 | 31.3 | 186.3 | 800.9 | 0.1 | 9.7 | 11 |
| 3 nodes, kafka | idle | app3 | 31.1 | 184.9 | 813 | 0.1 | 9.7 | 11 |
| 3 nodes, kafka | idle | db | 16 | 80.6 | 714.3 | – | – | – |
| 3 nodes, kafka | idle | kafka | 36.2 | 187.4 | 1028.1 | – | – | – |
| 3 nodes, kafka | reconnect_storm | app | 175.6 | 235.3 | 821.1 | 157 | 15 | 27 |
| 3 nodes, kafka | reconnect_storm | app2 | 175.3 | 228.8 | 870.7 | 157 | 15 | 27 |
| 3 nodes, kafka | reconnect_storm | app3 | 172.7 | 236.1 | 831.2 | 157 | 15 | 27 |
| 3 nodes, kafka | reconnect_storm | db | 168.7 | 328.9 | 763.6 | – | – | – |
| 3 nodes, kafka | reconnect_storm | kafka | 50.8 | 182.6 | 1036.3 | – | – | – |
| 3 nodes, kafka | rolling_restart | app | 217.2 | 347.8 | 1042.4 | 110.3 | 2.5 | 30 |
| 3 nodes, kafka | rolling_restart | app2 | 152.3 | 279.8 | 757 | 110.3 | 2.5 | 30 |
| 3 nodes, kafka | rolling_restart | app3 | 174.7 | 370.4 | 1202.2 | 110.3 | 2.5 | 30 |
| 3 nodes, kafka | rolling_restart | db | 175.9 | 357.8 | 799.5 | – | – | – |
| 3 nodes, kafka | rolling_restart | kafka | 47.3 | 192.9 | 1061.9 | – | – | – |
| 3 nodes, kafka | node_kill | app | 154.6 | 230.1 | 831.6 | 176.3 | 5.9 | 25 |
| 3 nodes, kafka | node_kill | app2 | 201.2 | 275.9 | 812.3 | 176.3 | 5.9 | 25 |
| 3 nodes, kafka | node_kill | app3 | 182 | 243.1 | 622 | 176.3 | 5.9 | 25 |
| 3 nodes, kafka | node_kill | db | 183.9 | 344.2 | 838.8 | – | – | – |
| 3 nodes, kafka | node_kill | kafka | 45.1 | 196.3 | 1069.1 | – | – | – |
Setup
- 1 node, native: 12 vCPU and 30.8 GiB visible to Docker (MINGW64_NT-10.0-26200, local, kernel 6.6.87.2-microsoft-standard-WSL2, Docker 29.8.2); per-node limits CPU unlimited, memory unlimited; Postgres 17.11, 217.1 MiB after the run; pool size 20 + 10 background; mock provider latency 20+10 ms, error rate 0.
e2e: conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=60webhook_burst: rate=500, duration_s=30, participants=200, targets=3idle: sockets=10000, sockets_per_vu=200, connect_rate=100, ramp_s=100, hold_s=60reconnect_storm: mode=storm, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=60, storm_at_s=15, storm_every_s=15rolling_restart: mode=restart, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=70, storm_at_s=null, storm_every_s=nullnode_kill: mode=restart, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=70, storm_at_s=null, storm_every_s=nullbackbone: backbone=native, subscriber=platform webhook (activity.created)
- 3 nodes, native: 12 vCPU and 30.8 GiB visible to Docker (MINGW64_NT-10.0-26200, local, kernel 6.6.87.2-microsoft-standard-WSL2, Docker 29.8.2); per-node limits CPU unlimited, memory unlimited; Postgres 17.11, 265.7 MiB after the run; pool size 20 + 10 background; mock provider latency 20+10 ms, error rate 0.
e2e: conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=60webhook_burst: rate=500, duration_s=30, participants=200, targets=3idle: sockets=10000, sockets_per_vu=200, connect_rate=100, ramp_s=100, hold_s=60reconnect_storm: mode=storm, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=60, storm_at_s=15, storm_every_s=15rolling_restart: mode=restart, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=170, storm_at_s=null, storm_every_s=nullnode_kill: mode=restart, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=70, storm_at_s=null, storm_every_s=nullbackbone: backbone=native, subscriber=platform webhook (activity.created)
- 1 node, kafka: 12 vCPU and 30.8 GiB visible to Docker (MINGW64_NT-10.0-26200, local, kernel 6.6.87.2-microsoft-standard-WSL2, Docker 29.8.2); per-node limits CPU unlimited, memory unlimited; Postgres 17.11, 197.2 MiB after the run; pool size 20 + 10 background; mock provider latency 20+10 ms, error rate 0.
e2e: conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=60webhook_burst: rate=500, duration_s=30, participants=200, targets=3idle: sockets=10000, sockets_per_vu=200, connect_rate=100, ramp_s=100, hold_s=60reconnect_storm: mode=storm, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=60, storm_at_s=15, storm_every_s=15rolling_restart: mode=restart, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=70, storm_at_s=null, storm_every_s=nullnode_kill: mode=restart, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=70, storm_at_s=null, storm_every_s=nullbackbone: backbone=kafka, subscriber=platform webhook (activity.created)
- 3 nodes, kafka: 12 vCPU and 30.8 GiB visible to Docker (MINGW64_NT-10.0-26200, local, kernel 6.6.87.2-microsoft-standard-WSL2, Docker 29.8.2); per-node limits CPU unlimited, memory unlimited; Postgres 17.11, 262.8 MiB after the run; pool size 20 + 10 background; mock provider latency 20+10 ms, error rate 0.
e2e: conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=60webhook_burst: rate=500, duration_s=30, participants=200, targets=3idle: sockets=10000, sockets_per_vu=200, connect_rate=100, ramp_s=100, hold_s=60reconnect_storm: mode=storm, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=60, storm_at_s=15, storm_every_s=15rolling_restart: mode=restart, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=170, storm_at_s=null, storm_every_s=nullnode_kill: mode=restart, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=70, storm_at_s=null, storm_every_s=nullbackbone: backbone=kafka, subscriber=platform webhook (activity.created)
Checks
- ✅ 1 node, native: e2e: k6 ran and correctness thresholds passed (k6 exit 0)
- ✅ 1 node, native: e2e: messages sent and received live (6000 acked, 60000 latency samples)
- ✅ 1 node, native: webhook_burst: k6 ran and correctness thresholds passed (k6 exit 99)
- ⚠️ 1 node, native: webhook_burst: performance targets (missed: http_req_duration{name:POST /inbound}: p(95)<500, dropped_iterations: count<1, inbound_rejected: count<1)
- ✅ 1 node, native: webhook_burst: every accepted message stored and delivered to every target (accepted 4284, stored 4284, delivered 12852/12852)
- ✅ 1 node, native: idle: k6 ran and correctness thresholds passed (k6 exit 0)
- ✅ 1 node, native: reconnect_storm: k6 ran and correctness thresholds passed (k6 exit 0)
- ✅ 1 node, native: reconnect_storm: messages sent and received live (5996 acked, 59607 latency samples)
- ✅ 1 node, native: rolling_restart: k6 ran and correctness thresholds passed (k6 exit 0)
- ✅ 1 node, native: rolling_restart: messages sent and received live (6740 acked, 64048 latency samples)
- ✅ 1 node, native: node_kill: k6 ran and correctness thresholds passed (k6 exit 0)
- ✅ 1 node, native: node_kill: messages sent and received live (6629 acked, 64623 latency samples)
- ✅ 1 node, native: backbone (native): every activity observed by the subscriber exactly once (31149/31149 observed, 31149 webhook events (0 duplicates), 1 HTTP redeliveries)
- ✅ 1 node, native: e2e: zero lost acked messages (lost 0, never acked 0)
- ✅ 1 node, native: reconnect_storm: zero lost acked messages (lost 0, never acked 0)
- ✅ 1 node, native: rolling_restart: zero lost acked messages (lost 0, never acked 0)
- ✅ 1 node, native: node_kill: zero lost acked messages (lost 0, never acked 0)
- ✅ 1 node, native: fan-out channel: every activity delivered to the outbound webhook (26865/26865 delivered, 0 jobs still open)
- ✅ 3 nodes, native: e2e: k6 ran and correctness thresholds passed (k6 exit 99)
- ⚠️ 3 nodes, native: e2e: performance targets (missed: e2e_latency_ms: p(95)<250)
- ✅ 3 nodes, native: e2e: messages sent and received live (6000 acked, 60000 latency samples)
- ✅ 3 nodes, native: webhook_burst: k6 ran and correctness thresholds passed (k6 exit 99)
- ⚠️ 3 nodes, native: webhook_burst: performance targets (missed: inbound_rejected: count<1, dropped_iterations: count<1, http_req_duration{name:POST /inbound}: p(95)<500)
- ✅ 3 nodes, native: webhook_burst: every accepted message stored and delivered to every target (accepted 3223, stored 3223, delivered 9669/9669)
- ✅ 3 nodes, native: idle: k6 ran and correctness thresholds passed (k6 exit 0)
- ✅ 3 nodes, native: reconnect_storm: k6 ran and correctness thresholds passed (k6 exit 0)
- ✅ 3 nodes, native: reconnect_storm: messages sent and received live (5986 acked, 59002 latency samples)
- ✅ 3 nodes, native: rolling_restart: k6 ran and correctness thresholds passed (k6 exit 0)
- ✅ 3 nodes, native: rolling_restart: messages sent and received live (16965 acked, 168393 latency samples)
- ✅ 3 nodes, native: node_kill: k6 ran and correctness thresholds passed (k6 exit 0)
- ✅ 3 nodes, native: node_kill: messages sent and received live (6955 acked, 69378 latency samples)
- ✅ 3 nodes, native: backbone (native): every activity observed by the subscriber exactly once (40629/40629 observed, 40629 webhook events (0 duplicates), 0 HTTP redeliveries)
- ✅ 3 nodes, native: e2e: zero lost acked messages (lost 0, never acked 0)
- ✅ 3 nodes, native: reconnect_storm: zero lost acked messages (lost 0, never acked 0)
- ✅ 3 nodes, native: rolling_restart: zero lost acked messages (lost 0, never acked 0)
- ✅ 3 nodes, native: node_kill: zero lost acked messages (lost 0, never acked 0)
- ✅ 3 nodes, native: fan-out channel: every activity delivered to the outbound webhook (37406/37406 delivered, 0 jobs still open)
- ✅ 1 node, kafka: e2e: k6 ran and correctness thresholds passed (k6 exit 0)
- ✅ 1 node, kafka: e2e: messages sent and received live (6000 acked, 60000 latency samples)
- ✅ 1 node, kafka: webhook_burst: k6 ran and correctness thresholds passed (k6 exit 99)
- ⚠️ 1 node, kafka: webhook_burst: performance targets (missed: inbound_rejected: count<1, dropped_iterations: count<1, http_req_duration{name:POST /inbound}: p(95)<500)
- ✅ 1 node, kafka: webhook_burst: every accepted message stored and delivered to every target (accepted 3840, stored 3840, delivered 11520/11520)
- ✅ 1 node, kafka: idle: k6 ran and correctness thresholds passed (k6 exit 0)
- ✅ 1 node, kafka: reconnect_storm: k6 ran and correctness thresholds passed (k6 exit 0)
- ✅ 1 node, kafka: reconnect_storm: messages sent and received live (5996 acked, 59270 latency samples)
- ✅ 1 node, kafka: rolling_restart: k6 ran and correctness thresholds passed (k6 exit 0)
- ✅ 1 node, kafka: rolling_restart: messages sent and received live (3766 acked, 30787 latency samples)
- ✅ 1 node, kafka: node_kill: k6 ran and correctness thresholds passed (k6 exit 0)
- ✅ 1 node, kafka: node_kill: messages sent and received live (6565 acked, 63995 latency samples)
- ✅ 1 node, kafka: backbone (kafka): every activity observed by the subscriber exactly once (27667/27667 observed, 27667 webhook events (0 duplicates), 3 HTTP redeliveries)
- ✅ 1 node, kafka: e2e: zero lost acked messages (lost 0, never acked 0)
- ✅ 1 node, kafka: reconnect_storm: zero lost acked messages (lost 0, never acked 0)
- ✅ 1 node, kafka: rolling_restart: zero lost acked messages (lost 0, never acked 0)
- ✅ 1 node, kafka: node_kill: zero lost acked messages (lost 0, never acked 0)
- ✅ 1 node, kafka: fan-out channel: every activity delivered to the outbound webhook (23827/23827 delivered, 0 jobs still open)
- ✅ 3 nodes, kafka: e2e: k6 ran and correctness thresholds passed (k6 exit 0)
- ✅ 3 nodes, kafka: e2e: messages sent and received live (6000 acked, 60000 latency samples)
- ✅ 3 nodes, kafka: webhook_burst: k6 ran and correctness thresholds passed (k6 exit 99)
- ⚠️ 3 nodes, kafka: webhook_burst: performance targets (missed: dropped_iterations: count<1, http_req_duration{name:POST /inbound}: p(95)<500, inbound_rejected: count<1)
- ✅ 3 nodes, kafka: webhook_burst: every accepted message stored and delivered to every target (accepted 2639, stored 2639, delivered 7917/7917)
- ✅ 3 nodes, kafka: idle: k6 ran and correctness thresholds passed (k6 exit 0)
- ✅ 3 nodes, kafka: reconnect_storm: k6 ran and correctness thresholds passed (k6 exit 0)
- ✅ 3 nodes, kafka: reconnect_storm: messages sent and received live (5976 acked, 58975 latency samples)
- ✅ 3 nodes, kafka: rolling_restart: k6 ran and correctness thresholds passed (k6 exit 0)
- ✅ 3 nodes, kafka: rolling_restart: messages sent and received live (16941 acked, 167765 latency samples)
- ✅ 3 nodes, kafka: node_kill: k6 ran and correctness thresholds passed (k6 exit 0)
- ✅ 3 nodes, kafka: node_kill: messages sent and received live (6917 acked, 68964 latency samples)
- ✅ 3 nodes, kafka: backbone (kafka): every activity observed by the subscriber exactly once (39973/39973 observed, 39973 webhook events (0 duplicates), 0 HTTP redeliveries)
- ✅ 3 nodes, kafka: e2e: zero lost acked messages (lost 0, never acked 0)
- ✅ 3 nodes, kafka: reconnect_storm: zero lost acked messages (lost 0, never acked 0)
- ✅ 3 nodes, kafka: rolling_restart: zero lost acked messages (lost 0, never acked 0)
- ✅ 3 nodes, kafka: node_kill: zero lost acked messages (lost 0, never acked 0)
- ✅ 3 nodes, kafka: fan-out channel: every activity delivered to the outbound webhook (37334/37334 delivered, 0 jobs still open)