Skip to main content

Backbone matrix

How the event backbone (ADR-0070) changes the numbers of the load harness: every scenario on the Native backbone (Phoenix.PubSub + Oban on Postgres) and on Kafka, with one and three nodes. The backbone rows are the backbone zero-loss check: every activity of the run is published to the backbone, and a platform webhook subscriber must observe each one exactly once.

Measured with load/matrix.sh on DESKTOP-Q8TVMOL (2026-10-14).

Commit e1c2862cfe81, 2026-10-14T22:11:16Z, profile full. Generated by load/run.sh (see Benchmarks).

ScenarioMetric1 node, native3 nodes, native1 node, kafka3 nodes, kafka
e2eREST→WS p50 (ms)532084177
e2eREST→WS p95 (ms)11434678144
e2eREST→WS p99 (ms)175394117178
e2eREST POST p95 (ms)113.3345.777.8143
e2esustained msgs/s (acked)100100100100
e2eWS frames/s delivered1000100010001000
e2eWS connections1000100010001000
webhook_burstinbound accepted/s142.8107.412888
webhook_burstinbound POST p95 (ms)1469.51803.11546.11662.5
webhook_burstinbound→provider p50 (ms)3727225373276239826
webhook_burstinbound→provider p95 (ms)46747290703656412653
webhook_burstinbound→provider p99 (ms)48297297163698513389
webhook_burstdeliveries/s161.6177.1164.7183.1
webhook_burstqueue drained after burst (s)46.520.936.69.2
idleidle sockets connected10000100001000010000
idleconnect p95 (ms)6767
idlememory per idle socket (KiB)108.7105.8109.9113.4
reconnect_stormreconnect p95 (ms)88710611545.11318
reconnect_stormreconnect max (ms)1046118017781499
reconnect_stormREST→WS p95 during storms (ms)233461355509
rolling_restartnode back to ready, max (s)10.313.740.643.1
rolling_restartclient reconnect max (ms)101542023491312685
node_killkilled node back to ready (s)5.17.95.710
node_killclient reconnect max (ms)1008090010337924
backboneactivity events observed / expected31149406292766739973
backboneduplicate events at the subscriber0000
backbonesend→backbone subscriber p50 (ms)24323403573823
backbonesend→backbone subscriber p95 (ms)12484114932816239689
backbonesend→backbone subscriber p99 (ms)13609167833022746839
backbonecaught up after the delivery queues drained (s)0.30.40.52.2
e2elost acked messages0000
webhook_burstlost acked messages0000
reconnect_stormlost acked messages0000
rolling_restartlost acked messages0000
node_killlost acked messages0000
node_killholes recovered with sync / replay_unavailable errors54 / 58325 / 39025 / 25420 / 464
backbonelost acked messages0000

Resources​

Highest values sampled while the scenario ran (CPU and RSS about every 2.5 s, the rest every second).

TopologyScenarioNodeCPU avg %CPU max %RSS max (MiB)DB pool wait max (ms)Background pool wait max (ms)Oban queue depth max
1 node, nativee2eapp391.7573.71428.58515.4136
1 node, nativee2edb151.8246304.7–––
1 node, nativewebhook_burstapp359581.6650.2449.916.115866
1 node, nativewebhook_burstdb256.8478.3367.8–––
1 node, nativeidleapp59.6435.31627.103400
1 node, nativeidledb14.441.8375–––
1 node, nativereconnect_stormapp377.4595.61626.198.310.1189
1 node, nativereconnect_stormdb143.1265.8394.7–––
1 node, nativerolling_restartapp377.35891610.86.89.1297
1 node, nativerolling_restartdb121.6213.8397.5–––
1 node, nativenode_killapp346.4548.71380.4168.427.4671
1 node, nativenode_killdb116.8372.6463.1–––
3 nodes, nativee2eapp160.9216.3543.96.727.1377
3 nodes, nativee2eapp2150.3205.5623.66.727.1377
3 nodes, nativee2eapp3163.8197.9581.76.727.1377
3 nodes, nativee2edb169241.2617.6–––
3 nodes, nativewebhook_burstapp183.5216.6456.9417.155.911523
3 nodes, nativewebhook_burstapp2183.6235.4461.7417.155.911523
3 nodes, nativewebhook_burstapp3187.7241.5468417.155.911523
3 nodes, nativewebhook_burstdb306.2422.7707.4–––
3 nodes, nativeidleapp2671.4781.40.1223.30
3 nodes, nativeidleapp226.768.8797.90.1223.30
3 nodes, nativeidleapp325.172.6820.60.1223.30
3 nodes, nativeidledb15.545.8718.2–––
3 nodes, nativereconnect_stormapp160.8232.3809.3139.335.8451
3 nodes, nativereconnect_stormapp2154.5212.8827.5139.335.8451
3 nodes, nativereconnect_stormapp3160.4246.5904.9139.335.8451
3 nodes, nativereconnect_stormdb155291769.2–––
3 nodes, nativerolling_restartapp177.9347.21123.3168.154.31061
3 nodes, nativerolling_restartapp2153.1286.1774.1168.154.31061
3 nodes, nativerolling_restartapp3153.8305.21216.5168.154.31061
3 nodes, nativerolling_restartdb164302.8767.7–––
3 nodes, nativenode_killapp131.1219.4754206.819.7123
3 nodes, nativenode_killapp2176.1293.1734.2206.819.7123
3 nodes, nativenode_killapp3152.8239.3666.4206.819.7123
3 nodes, nativenode_killdb168.2373.1851.4–––
1 node, kafkae2eapp438.7544.3883.450.720
1 node, kafkae2edb154.3231.9292.8–––
1 node, kafkae2ekafka57.1214.7899.4–––
1 node, kafkawebhook_burstapp365.6477.3622.9324.12.210266
1 node, kafkawebhook_burstdb308496.9354–––
1 node, kafkawebhook_burstkafka45.1177.91070.1–––
1 node, kafkaidleapp69.2444.816640.11180
1 node, kafkaidledb14.354359.1–––
1 node, kafkaidlekafka31.5170.31076.2–––
1 node, kafkareconnect_stormapp413.9572.41700.9106.12.774
1 node, kafkareconnect_stormdb171.7318.1390.6–––
1 node, kafkareconnect_stormkafka46.71811085.4–––
1 node, kafkarolling_restartapp239.6536.41735.7136.11.4211
1 node, kafkarolling_restartdb81.6237.3424.1–––
1 node, kafkarolling_restartkafka35.4193.11093.6–––
1 node, kafkanode_killapp374.35481379.3953.197
1 node, kafkanode_killdb130.1331.9438–––
1 node, kafkanode_killkafka45.9197.91109–––
3 nodes, kafkae2eapp186.1222.3601.416.71.117
3 nodes, kafkae2eapp2184.2221.5522.116.71.117
3 nodes, kafkae2eapp3178.9221.6491.216.71.117
3 nodes, kafkae2edb193.5280.1618–––
3 nodes, kafkae2ekafka58.2179.6811.2–––
3 nodes, kafkawebhook_burstapp194.7220.6484.4375.511.24000
3 nodes, kafkawebhook_burstapp2192.3218.2469.7375.511.24000
3 nodes, kafkawebhook_burstapp3199.3240.9474.4375.511.24000
3 nodes, kafkawebhook_burstdb365.4439.7705.1–––
3 nodes, kafkawebhook_burstkafka46.4140.4968.3–––
3 nodes, kafkaidleapp29.9193.4828.10.19.711
3 nodes, kafkaidleapp231.3186.3800.90.19.711
3 nodes, kafkaidleapp331.1184.98130.19.711
3 nodes, kafkaidledb1680.6714.3–––
3 nodes, kafkaidlekafka36.2187.41028.1–––
3 nodes, kafkareconnect_stormapp175.6235.3821.11571527
3 nodes, kafkareconnect_stormapp2175.3228.8870.71571527
3 nodes, kafkareconnect_stormapp3172.7236.1831.21571527
3 nodes, kafkareconnect_stormdb168.7328.9763.6–––
3 nodes, kafkareconnect_stormkafka50.8182.61036.3–––
3 nodes, kafkarolling_restartapp217.2347.81042.4110.32.530
3 nodes, kafkarolling_restartapp2152.3279.8757110.32.530
3 nodes, kafkarolling_restartapp3174.7370.41202.2110.32.530
3 nodes, kafkarolling_restartdb175.9357.8799.5–––
3 nodes, kafkarolling_restartkafka47.3192.91061.9–––
3 nodes, kafkanode_killapp154.6230.1831.6176.35.925
3 nodes, kafkanode_killapp2201.2275.9812.3176.35.925
3 nodes, kafkanode_killapp3182243.1622176.35.925
3 nodes, kafkanode_killdb183.9344.2838.8–––
3 nodes, kafkanode_killkafka45.1196.31069.1–––

Setup​

  • 1 node, native: 12 vCPU and 30.8 GiB visible to Docker (MINGW64_NT-10.0-26200, local, kernel 6.6.87.2-microsoft-standard-WSL2, Docker 29.8.2); per-node limits CPU unlimited, memory unlimited; Postgres 17.11, 217.1 MiB after the run; pool size 20 + 10 background; mock provider latency 20+10 ms, error rate 0.
    • e2e: conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=60
    • webhook_burst: rate=500, duration_s=30, participants=200, targets=3
    • idle: sockets=10000, sockets_per_vu=200, connect_rate=100, ramp_s=100, hold_s=60
    • reconnect_storm: mode=storm, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=60, storm_at_s=15, storm_every_s=15
    • rolling_restart: mode=restart, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=70, storm_at_s=null, storm_every_s=null
    • node_kill: mode=restart, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=70, storm_at_s=null, storm_every_s=null
    • backbone: backbone=native, subscriber=platform webhook (activity.created)
  • 3 nodes, native: 12 vCPU and 30.8 GiB visible to Docker (MINGW64_NT-10.0-26200, local, kernel 6.6.87.2-microsoft-standard-WSL2, Docker 29.8.2); per-node limits CPU unlimited, memory unlimited; Postgres 17.11, 265.7 MiB after the run; pool size 20 + 10 background; mock provider latency 20+10 ms, error rate 0.
    • e2e: conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=60
    • webhook_burst: rate=500, duration_s=30, participants=200, targets=3
    • idle: sockets=10000, sockets_per_vu=200, connect_rate=100, ramp_s=100, hold_s=60
    • reconnect_storm: mode=storm, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=60, storm_at_s=15, storm_every_s=15
    • rolling_restart: mode=restart, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=170, storm_at_s=null, storm_every_s=null
    • node_kill: mode=restart, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=70, storm_at_s=null, storm_every_s=null
    • backbone: backbone=native, subscriber=platform webhook (activity.created)
  • 1 node, kafka: 12 vCPU and 30.8 GiB visible to Docker (MINGW64_NT-10.0-26200, local, kernel 6.6.87.2-microsoft-standard-WSL2, Docker 29.8.2); per-node limits CPU unlimited, memory unlimited; Postgres 17.11, 197.2 MiB after the run; pool size 20 + 10 background; mock provider latency 20+10 ms, error rate 0.
    • e2e: conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=60
    • webhook_burst: rate=500, duration_s=30, participants=200, targets=3
    • idle: sockets=10000, sockets_per_vu=200, connect_rate=100, ramp_s=100, hold_s=60
    • reconnect_storm: mode=storm, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=60, storm_at_s=15, storm_every_s=15
    • rolling_restart: mode=restart, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=70, storm_at_s=null, storm_every_s=null
    • node_kill: mode=restart, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=70, storm_at_s=null, storm_every_s=null
    • backbone: backbone=kafka, subscriber=platform webhook (activity.created)
  • 3 nodes, kafka: 12 vCPU and 30.8 GiB visible to Docker (MINGW64_NT-10.0-26200, local, kernel 6.6.87.2-microsoft-standard-WSL2, Docker 29.8.2); per-node limits CPU unlimited, memory unlimited; Postgres 17.11, 262.8 MiB after the run; pool size 20 + 10 background; mock provider latency 20+10 ms, error rate 0.
    • e2e: conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=60
    • webhook_burst: rate=500, duration_s=30, participants=200, targets=3
    • idle: sockets=10000, sockets_per_vu=200, connect_rate=100, ramp_s=100, hold_s=60
    • reconnect_storm: mode=storm, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=60, storm_at_s=15, storm_every_s=15
    • rolling_restart: mode=restart, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=170, storm_at_s=null, storm_every_s=null
    • node_kill: mode=restart, conversations=100, subscribers_per_conversation=10, sockets=1000, rate_per_conversation=1, duration_s=70, storm_at_s=null, storm_every_s=null
    • backbone: backbone=kafka, subscriber=platform webhook (activity.created)

Checks​

  • ✅ 1 node, native: e2e: k6 ran and correctness thresholds passed (k6 exit 0)
  • ✅ 1 node, native: e2e: messages sent and received live (6000 acked, 60000 latency samples)
  • ✅ 1 node, native: webhook_burst: k6 ran and correctness thresholds passed (k6 exit 99)
  • ⚠️ 1 node, native: webhook_burst: performance targets (missed: http_req_duration{name:POST /inbound}: p(95)<500, dropped_iterations: count<1, inbound_rejected: count<1)
  • ✅ 1 node, native: webhook_burst: every accepted message stored and delivered to every target (accepted 4284, stored 4284, delivered 12852/12852)
  • ✅ 1 node, native: idle: k6 ran and correctness thresholds passed (k6 exit 0)
  • ✅ 1 node, native: reconnect_storm: k6 ran and correctness thresholds passed (k6 exit 0)
  • ✅ 1 node, native: reconnect_storm: messages sent and received live (5996 acked, 59607 latency samples)
  • ✅ 1 node, native: rolling_restart: k6 ran and correctness thresholds passed (k6 exit 0)
  • ✅ 1 node, native: rolling_restart: messages sent and received live (6740 acked, 64048 latency samples)
  • ✅ 1 node, native: node_kill: k6 ran and correctness thresholds passed (k6 exit 0)
  • ✅ 1 node, native: node_kill: messages sent and received live (6629 acked, 64623 latency samples)
  • ✅ 1 node, native: backbone (native): every activity observed by the subscriber exactly once (31149/31149 observed, 31149 webhook events (0 duplicates), 1 HTTP redeliveries)
  • ✅ 1 node, native: e2e: zero lost acked messages (lost 0, never acked 0)
  • ✅ 1 node, native: reconnect_storm: zero lost acked messages (lost 0, never acked 0)
  • ✅ 1 node, native: rolling_restart: zero lost acked messages (lost 0, never acked 0)
  • ✅ 1 node, native: node_kill: zero lost acked messages (lost 0, never acked 0)
  • ✅ 1 node, native: fan-out channel: every activity delivered to the outbound webhook (26865/26865 delivered, 0 jobs still open)
  • ✅ 3 nodes, native: e2e: k6 ran and correctness thresholds passed (k6 exit 99)
  • ⚠️ 3 nodes, native: e2e: performance targets (missed: e2e_latency_ms: p(95)<250)
  • ✅ 3 nodes, native: e2e: messages sent and received live (6000 acked, 60000 latency samples)
  • ✅ 3 nodes, native: webhook_burst: k6 ran and correctness thresholds passed (k6 exit 99)
  • ⚠️ 3 nodes, native: webhook_burst: performance targets (missed: inbound_rejected: count<1, dropped_iterations: count<1, http_req_duration{name:POST /inbound}: p(95)<500)
  • ✅ 3 nodes, native: webhook_burst: every accepted message stored and delivered to every target (accepted 3223, stored 3223, delivered 9669/9669)
  • ✅ 3 nodes, native: idle: k6 ran and correctness thresholds passed (k6 exit 0)
  • ✅ 3 nodes, native: reconnect_storm: k6 ran and correctness thresholds passed (k6 exit 0)
  • ✅ 3 nodes, native: reconnect_storm: messages sent and received live (5986 acked, 59002 latency samples)
  • ✅ 3 nodes, native: rolling_restart: k6 ran and correctness thresholds passed (k6 exit 0)
  • ✅ 3 nodes, native: rolling_restart: messages sent and received live (16965 acked, 168393 latency samples)
  • ✅ 3 nodes, native: node_kill: k6 ran and correctness thresholds passed (k6 exit 0)
  • ✅ 3 nodes, native: node_kill: messages sent and received live (6955 acked, 69378 latency samples)
  • ✅ 3 nodes, native: backbone (native): every activity observed by the subscriber exactly once (40629/40629 observed, 40629 webhook events (0 duplicates), 0 HTTP redeliveries)
  • ✅ 3 nodes, native: e2e: zero lost acked messages (lost 0, never acked 0)
  • ✅ 3 nodes, native: reconnect_storm: zero lost acked messages (lost 0, never acked 0)
  • ✅ 3 nodes, native: rolling_restart: zero lost acked messages (lost 0, never acked 0)
  • ✅ 3 nodes, native: node_kill: zero lost acked messages (lost 0, never acked 0)
  • ✅ 3 nodes, native: fan-out channel: every activity delivered to the outbound webhook (37406/37406 delivered, 0 jobs still open)
  • ✅ 1 node, kafka: e2e: k6 ran and correctness thresholds passed (k6 exit 0)
  • ✅ 1 node, kafka: e2e: messages sent and received live (6000 acked, 60000 latency samples)
  • ✅ 1 node, kafka: webhook_burst: k6 ran and correctness thresholds passed (k6 exit 99)
  • ⚠️ 1 node, kafka: webhook_burst: performance targets (missed: inbound_rejected: count<1, dropped_iterations: count<1, http_req_duration{name:POST /inbound}: p(95)<500)
  • ✅ 1 node, kafka: webhook_burst: every accepted message stored and delivered to every target (accepted 3840, stored 3840, delivered 11520/11520)
  • ✅ 1 node, kafka: idle: k6 ran and correctness thresholds passed (k6 exit 0)
  • ✅ 1 node, kafka: reconnect_storm: k6 ran and correctness thresholds passed (k6 exit 0)
  • ✅ 1 node, kafka: reconnect_storm: messages sent and received live (5996 acked, 59270 latency samples)
  • ✅ 1 node, kafka: rolling_restart: k6 ran and correctness thresholds passed (k6 exit 0)
  • ✅ 1 node, kafka: rolling_restart: messages sent and received live (3766 acked, 30787 latency samples)
  • ✅ 1 node, kafka: node_kill: k6 ran and correctness thresholds passed (k6 exit 0)
  • ✅ 1 node, kafka: node_kill: messages sent and received live (6565 acked, 63995 latency samples)
  • ✅ 1 node, kafka: backbone (kafka): every activity observed by the subscriber exactly once (27667/27667 observed, 27667 webhook events (0 duplicates), 3 HTTP redeliveries)
  • ✅ 1 node, kafka: e2e: zero lost acked messages (lost 0, never acked 0)
  • ✅ 1 node, kafka: reconnect_storm: zero lost acked messages (lost 0, never acked 0)
  • ✅ 1 node, kafka: rolling_restart: zero lost acked messages (lost 0, never acked 0)
  • ✅ 1 node, kafka: node_kill: zero lost acked messages (lost 0, never acked 0)
  • ✅ 1 node, kafka: fan-out channel: every activity delivered to the outbound webhook (23827/23827 delivered, 0 jobs still open)
  • ✅ 3 nodes, kafka: e2e: k6 ran and correctness thresholds passed (k6 exit 0)
  • ✅ 3 nodes, kafka: e2e: messages sent and received live (6000 acked, 60000 latency samples)
  • ✅ 3 nodes, kafka: webhook_burst: k6 ran and correctness thresholds passed (k6 exit 99)
  • ⚠️ 3 nodes, kafka: webhook_burst: performance targets (missed: dropped_iterations: count<1, http_req_duration{name:POST /inbound}: p(95)<500, inbound_rejected: count<1)
  • ✅ 3 nodes, kafka: webhook_burst: every accepted message stored and delivered to every target (accepted 2639, stored 2639, delivered 7917/7917)
  • ✅ 3 nodes, kafka: idle: k6 ran and correctness thresholds passed (k6 exit 0)
  • ✅ 3 nodes, kafka: reconnect_storm: k6 ran and correctness thresholds passed (k6 exit 0)
  • ✅ 3 nodes, kafka: reconnect_storm: messages sent and received live (5976 acked, 58975 latency samples)
  • ✅ 3 nodes, kafka: rolling_restart: k6 ran and correctness thresholds passed (k6 exit 0)
  • ✅ 3 nodes, kafka: rolling_restart: messages sent and received live (16941 acked, 167765 latency samples)
  • ✅ 3 nodes, kafka: node_kill: k6 ran and correctness thresholds passed (k6 exit 0)
  • ✅ 3 nodes, kafka: node_kill: messages sent and received live (6917 acked, 68964 latency samples)
  • ✅ 3 nodes, kafka: backbone (kafka): every activity observed by the subscriber exactly once (39973/39973 observed, 39973 webhook events (0 duplicates), 0 HTTP redeliveries)
  • ✅ 3 nodes, kafka: e2e: zero lost acked messages (lost 0, never acked 0)
  • ✅ 3 nodes, kafka: reconnect_storm: zero lost acked messages (lost 0, never acked 0)
  • ✅ 3 nodes, kafka: rolling_restart: zero lost acked messages (lost 0, never acked 0)
  • ✅ 3 nodes, kafka: node_kill: zero lost acked messages (lost 0, never acked 0)
  • ✅ 3 nodes, kafka: fan-out channel: every activity delivered to the outbound webhook (37334/37334 delivered, 0 jobs still open)