Benchmarks and load testing

Converger's performance numbers come from a load harness that drives the production release over HTTP and WebSocket, the same way clients and providers do. Each run records the hardware it ran on, and it fails if any message the server acknowledged is lost. The decision and its trade-offs are in ADR-0049.

Published results

Run	Topologies	Notes
2026-10-12, 612ec15	1 node and 3 nodes, native backbone	First run of the harness, on a developer machine (Docker Desktop). Before #35.
2026-10-15, a8dd704	3 nodes, native backbone, rolling_restart only	10 000 sockets through a rolling restart (#190): zero lost acked messages; reconnect p95 93.6 s, tracked in #198.
2026-10-15, after #198	3 nodes, native backbone, rolling_restart only	The same 10 000-socket rolling restart with the admission queue (ADR-0077): reconnect p95 6.0 s (was 93.6 s), zero lost acked messages.
2026-10-15, partitions at 100M rows	Postgres + MinIO, test/load/partition_100m/run.sh	Oldest month of 100M rows archived, verified and dropped while writers inserted (#192).
Backbone matrix	Native and Kafka backbones x 1 and 3 nodes	Regenerated by the nightly Load workflow. Pending: no numbers published yet (see the page).
Dead-channel isolation	1 node, native backbone	Epic #59 exit criterion: a healthy tenant's delivery latency with and without another tenant's dead channel and a 10,000-delivery backlog.
Partition retention at scale	Postgres only	Epic #59 exit criterion: detach, archive, verify and drop of one month at 10M and 30M rows per table; 100M per table extrapolated. A 100M-row dataset is measured in 2026-10-15, partitions at 100M rows.

The nightly Load workflow publishes a report for every run as its job summary and as the load-results-* artifacts. Pages here are the runs worth keeping: a new baseline, a release, or a before/after comparison.

:::note Status The backbone dimension (#66) has two values in the harness: native (Phoenix.PubSub over Erlang distribution plus Oban on Postgres) and kafka; the backbone matrix compares them on 1 and 3 nodes. NATS JetStream and RabbitMQ backbones exist in Converger but are not wired into the harness yet. The scale work (#29, #30, #31, #35) has merged, and the first published run predates #35. The after-#35 run with before/after columns is still to be published: the first attempt shared the machine with unrelated workloads, so its numbers were not comparable. The harness itself was checked against #35: under database pressure the server answers replay_unavailable, the subscribers recover every hole with sync, and no acknowledged message is lost. :::

Scenarios

Scenario	Script	What it does	Main numbers
e2e	load/e2e.js	One VU per conversation opens SUBSCRIBERS Protocol v1 sockets on it, then POSTs RATE activities per second over the tenant REST API.	REST → WebSocket p50/p95/p99, REST POST p95, sustained acked msgs/s, frames/s delivered, connections
webhook_burst	load/webhook_burst.js	Signed generic inbound webhooks at BURST_RATE/s for BURST_S s from 200 external users. A routing rule fans every message out to three outbound webhook channels on the mock provider.	inbound accepted/s and POST p95, inbound → provider p50/p95/p99 (slowest target), deliveries/s, time for the queue to drain
idle	load/idle.js	Opens SOCKETS connections (10 000) at CONNECT_RATE/s and holds them idle (a ping every 25 s).	connections, connect p95, RSS growth and memory per idle socket
reconnect_storm	load/reconnect.js (MODE=storm)	e2e traffic, and every 15 s every socket drops and reconnects at once with its last seq as the watermark.	reconnect (close → welcome) p95/max, REST → WebSocket p95 during storms
rolling_restart	load/reconnect.js (MODE=restart)	e2e traffic while run.sh restarts each node in turn with SIGTERM (readiness flips, sockets drain, the node comes back). Clients reconnect to a random other node (as a load balancer spreads them), after a jittered backoff capped at 5 s, or the retryAfterMs of an unavailable error if longer (#198; k6 does not expose the 1012 close reason, so the drain's hint is not seen).	time until each node is ready again, client reconnect max
node_kill	load/reconnect.js (MODE=restart)	e2e traffic while run.sh SIGKILLs one node and starts it again.	time until the killed node is ready, client reconnect max
dead_channel (opt-in)	load/dead_channel.js	Two phases back to back: a healthy tenant posts LOAD_DEAD_RATE messages/s on each of LOAD_DEAD_CONVERSATIONS conversations to an outbound webhook (baseline), then again while another tenant's webhook channel, which never answers (/hook/hang-dead), gets a backlog of LOAD_DEAD_BACKLOG deliveries. Both tenants are on the default tier unless LOAD_DEAD_TIER says otherwise. See Dead-channel isolation.	healthy tenant send → provider p50/p95/p99 per phase, breaker open time, backlog parked time, requests that reached the dead provider

Every scenario that sends messages also runs the zero-loss check. Senders retry each message with the same idempotency key until it is acknowledged, so a restart delays messages but must not lose them.

Metrics

Metric	Source
End-to-end latency p50/p95/p99 (ms)	REST and inbound texts carry their send time (load <unix ms> <key>); subscriber sockets and the mock provider subtract it from the arrival time. All clients run on the same host.
Sustained msgs/s	Messages acknowledged by the server divided by the sending time.
WebSocket connections	welcome frames received (ws_connected).
CPU and RSS per node	docker stats, streamed about once a second for every node and Postgres.
DB pool wait	The converger_repo_query_queue_time histogram (time waiting for a pool connection) from each node's /metrics, read every second: the mean wait of the queries in each interval. The report shows the highest interval mean. Runs before #33 (such as the first published one) used the last queue time instead.
Oban queue depth	Open jobs (available, scheduled, executing, retryable) in oban_jobs, every second.
Recovery time after a restart or kill (s)	From the restart or SIGKILL until GET /health/ready answers 200 again.
Reconnect time (ms)	From a socket closing to the welcome of its next connection.
Lost acked messages	See below. Must be 0.
Hardware	vCPUs and memory visible to Docker, OS, kernel, Docker and Postgres versions, per-node CPU and memory limits, pool size, database size after the run.

Zero-loss check

Each subscriber socket only reads. Because seq is allocated gap-free per conversation (ADR-0006), and the server fills gaps and drops duplicates on every connection (Protocol v1, section 6.3), a correct server delivers every seq exactly once and in order.

Under database pressure the server may be unable to replay or fill a gap. It then keeps the connection, sends error replay_unavailable and continues live (Database pools and pressure). The subscriber does what a client must do: it keeps the last contiguous seq, holds frames received beyond a hole, and sends sync from the contiguous position after retryAfterMs (or one second after it notices a hole). When the sending stops, each subscriber sends a ping and reads the conversation head (headSeq) from the heartbeat. While it is behind the head it syncs and asks again, for up to 60 s. The harness counts:

• lost_acked_messages: headSeq minus the last contiguous seq at the end, so every message that never arrived;
• seq_duplicates: a frame delivered twice on the same connection that did not ask for a replay with sync. The protocol promises each frame once per connection from hello; repeats across a reconnect or after a sync are expected, and frames still arriving on a replaced connection are ignored;
• subscribers_unverified: subscribers that could not reach the server for the final check;
• for information, seq_gaps_resynced (holes the client recovered with sync) and replay_unavailable (the server's errors). They show the server under pressure, not loss.

Reconnects resume with hello.watermark set to the last contiguous seq, so a socket that was disconnected for a restart must still end with every message. For webhook deliveries, the harness compares the inbound messages accepted (2xx) with the activities stored in Postgres and with what the mock provider received on each target channel. It also checks that every REST activity reached the fan-out channel's outbound webhook.

Backbone zero-loss check

The checks above follow messages to WebSocket subscribers and channel deliveries. The event backbone (#66) has a check of its own: load/setup.exs subscribes a platform event webhook to activity.created on the mock provider (/hook/backbone), so every activity of the run (warm-up, REST, inbound burst, the restart scenarios) is published to the backbone and must reach that subscriber. After the delivery queues drained, run.sh waits until the subscriber has caught up, then the report compares:

• the activities the tenant stored with the distinct events the mock provider received (lost_acked_messages of the backbone rows: must be 0);
• the subscriber's webhook_events rows with their distinct backbone event ids (duplicate events: must be 0, the idempotency receipts make a redelivered event a no-op);
• for information, HTTP redeliveries (a webhook POST repeated after a node kill: at least once by design, receivers de-duplicate by event id) and the send → subscriber latency.

Because the check runs during rolling_restart and node_kill, it covers consumers killed and restarted mid-stream on every backbone. The extra webhook per activity is load the published runs before #66 did not have.

Running it

Requirements: Docker with Compose 2.24 or newer, bash, curl and openssl. These are present in Git Bash on Windows and on GitHub's Ubuntu runners. Nothing else needs to be installed: k6, the mock provider and the report run in containers.

load/run.sh                                  # 1 node, every scenario, full sizes
LOAD_NODES=3 load/run.sh                     # three clustered nodes (gossip)
LOAD_PROFILE=smoke load/run.sh               # small sizes, about 8 minutes
LOAD_SCENARIOS="e2e idle" LOAD_SKIP_BUILD=1 load/run.sh
LOAD_BACKBONE=kafka load/run.sh              # Kafka as the event backbone (adds a broker container)
LOAD_SCENARIOS=dead_channel load/run.sh      # dead-channel isolation (not in the default list)
load/matrix.sh                               # every backbone x node count, one table (below)

The script:

1. Writes throwaway secrets to load/.run/.env (gitignored; nothing is reused between runs).
2. Builds the release image converger-load:local from the Dockerfile and starts Postgres, the migration step, the node(s) and the mock provider as compose project converger-load. Nodes listen on 127.0.0.1:14100 (and 14101, 14102). Postgres is not published, so the harness runs next to a development stack.
3. Creates the fixtures on a node with bin/converger rpc (load/setup.exs). It makes a tenant with raised rate limits, an outbound webhook channel with the e2e conversations and one socket token per subscriber, the idle conversations, the ingress channel with a routing rule to the target channels, and the platform webhook subscription of the backbone zero-loss check.
4. Starts the sampler and runs each scenario with grafana/k6:1.8.1. For rolling_restart and node_kill it restarts or kills nodes meanwhile.
5. Waits until no delivery job is open and the backbone subscriber has caught up, collects the mock provider's statistics and the database counts, records the hardware, and writes load/.run/out/results.json and report.md (load/results.js).
6. Tears the stack down, volumes included (LOAD_KEEP=1 keeps it).

The exit code is non-zero when a loss check or a correctness threshold failed, or k6 crashed (see thresholds); missed performance targets are only reported. The output directory also holds each scenario's k6 log and summary JSON, samples.csv, mock_stats.json, the node logs and setup.json.

Tunables

Variable	full	smoke	Meaning
LOAD_NODES	1		1 or 3 nodes.
LOAD_BACKBONE	native		The event backbone: native or kafka (a single-node KRaft broker joins the stack and every node gets BACKBONE=kafka, see load/docker-compose.load-kafka*.yml). nats and rabbitmq are not wired into the harness yet.
LOAD_KAFKA_PARTITIONS	12		Partitions of the backbone topics with LOAD_BACKBONE=kafka.
LOAD_SCENARIOS	all six		Space-separated subset, in order.
LOAD_CONVERSATIONS / LOAD_SUBSCRIBERS	100 / 10	10 / 3	e2e and reconnect conversations, and sockets per conversation (1 000 sockets by default).
LOAD_RATE	1	2	REST messages per second per conversation.
LOAD_DURATION_S	60	20	Sending time of e2e and reconnect_storm. The restart scenarios run long enough for every restart.
LOAD_BURST_RATE / LOAD_BURST_S	500 / 30	50 / 10	Inbound webhook burst.
LOAD_TARGETS	3		Outbound channels the burst fans out to.
LOAD_IDLE_SOCKETS / LOAD_IDLE_HOLD_S	10000 / 60	500 / 15	Idle connections and how long they are held.
LOAD_IDLE_CONVERSATIONS	1000	50	Conversations the idle sockets share.
LOAD_CONNECT_RATE	100		New idle connections per second. Every handshake reads the database, so at much higher rates the pool, not memory, is what is measured.
LOAD_RESTART_GAP_S	10		Pause between node restarts.
LOAD_RESTART_DELAY_S	20		Seconds of traffic in rolling_restart and node_kill before the first restart.
LOAD_RESTART_CONNECT_RATE	0		Sockets per second at which the restart scenarios connect; 0 opens them all at once. The 10 000-socket rolling restart (#190) uses 100 with a 130 s delay, so every socket is in before the first restart.
LOAD_APP_CPUS / LOAD_APP_MEMORY	unlimited		Per-node limits (2, 2g) for capacity runs.
LOAD_POOL_SIZE / LOAD_BACKGROUND_POOL_SIZE	20 / 10		POOL_SIZE and BACKGROUND_POOL_SIZE of each node (Database pools).
MOCK_LATENCY_MS / MOCK_JITTER_MS / MOCK_ERROR_RATE	20 / 10 / 0		Mock provider answer time and the fraction of requests it fails with 500.
LOAD_SKIP_BUILD / LOAD_KEEP	0		Reuse the image; keep the stack after the run.
LOAD_IMAGE / LOAD_PROJECT	converger-load:local / converger-load		Image tag and compose project, so two checkouts can run the harness side by side (with their own LOAD_APP_PORTs).

The load stack sets OBAN_LIFELINE_RESCUE_AFTER_SECONDS=30 (LOAD_LIFELINE_RESCUE_AFTER_SECONDS), as the chaos test does. Otherwise deliveries that were executing on a SIGKILLed node would wait for the 30-minute default before they are retried.

Running a scenario with k6 directly

Against a stack you already run (for example LOAD_KEEP=1), with fixtures from load/setup.exs:

k6 run -e SETUP_JSON=load/.run/out/setup.json -e APP_URLS=http://127.0.0.1:14100 -e OUT_DIR=load/.run/out load/e2e.js

k6 prints the end-to-end p95 (e2e_latency_ms), the connections (ws_connected) and the loss counters, and exits with 99 when a threshold is crossed. Each script lists its variables at the top.

The thresholds are of two kinds:

• Correctness (lost_acked_messages, seq_duplicates, messages_never_acked, subscribers_unverified, and for idle, ws_connected and ws_unexpected_closes): a miss fails the run.
• Performance targets from the PRD and the scenario sizes: REST → WebSocket p95 below E2E_P95_MS (250), REST POST p95 below REST_P95_MS (1000), inbound POST p95 below 500 ms, no burst iteration dropped and no inbound request rejected. A miss is reported as "⚠️ performance targets" in the report, but does not fail the run. The baseline comparison is what gates performance, so the nightly job stays green while a known gap (such as the 250 ms target today) is open and still catches regressions.

The mock provider

load/mock_provider/server.js stands in for every endpoint Converger delivers to (it has no dependencies and runs in node:22-alpine). It answers POST /hook/<path> after MOCK_LATENCY_MS plus up to MOCK_JITTER_MS, fails a MOCK_ERROR_RATE fraction of requests with 500 (Converger retries them), and records per path the requests, failures, unique deliveries, duplicates and the latency of each first successful delivery. It serves them at GET /stats and clears them with POST /reset.

Nightly job and regression gate

.github/workflows/load.yml runs every night at 02:00 UTC, on demand (workflow_dispatch, with a profile input), and on pull requests that change load/. It runs the full profile on 1 and 3 nodes on ubuntu-24.04 runners, with the Native and the Kafka backbone, writes each report to the job summary and uploads load/.run/out/ as load-results-<backbone>-<n>node (the backbone matrix job then renders them together).

Each run is compared with load/baselines/ubuntu-24.04-<backbone>-<n>node.json (the results.json of an accepted run on the same runner image):

node load/report.js --baseline load/baselines/ubuntu-24.04-native-1node.json load/.run/out/results.json

The job fails when:

• any run has a lost acked message, a failed loss check, a failed correctness threshold or a k6 crash, or
• a gated metric got worse by more than 15 % and by more than its noise floor. The gated metrics are the REST → WebSocket p95, REST POST p95, inbound POST p95, inbound → provider p95, reconnect p95 and memory per idle socket. The noise floor is an absolute change below which the metric counts as unchanged: 5 ms for most latencies, 10 ms for delivery, 50 ms for reconnects, 2 KiB for memory.

With no baseline for the runner, the comparison is skipped with a notice. To accept new numbers (other hardware, or a deliberate trade-off), download results.json from a nightly artifact and commit it as the baseline, saying why in the pull request.

GitHub-hosted runners are shared machines. k6, Postgres and every node also share the runner's 4 vCPUs. Treat the nightly numbers as a regression signal and use dedicated hardware with LOAD_APP_CPUS for capacity planning.

Publishing a run

load/report.js merges several runs (one per topology) into one page, optionally with a baseline column:

LOAD_NODES=1 load/run.sh && cp load/.run/out/results.json load/results/<date>-<commit>-1node.json
LOAD_NODES=3 load/run.sh && cp load/.run/out/results.json load/results/<date>-<commit>-3node.json
node load/report.js --out docs/benchmarks/<date>-<commit>.md \
  load/results/<date>-<commit>-1node.json load/results/<date>-<commit>-3node.json

Add front matter, the hardware context and the findings to the page, a row to Published results, and update the summary table in the README. Keep the results.json files in load/results/: a later run shows its before/after columns against them, for example after #35:

node load/report.js --baseline load/results/2026-10-12-612ec15-1node.json \
  --baseline load/results/2026-10-12-612ec15-3node.json \
  load/results/<new>-1node.json load/results/<new>-3node.json

Compare runs from the same machine only; the report prints both hardware records.

Backbone matrix

load/matrix.sh runs load/run.sh for every backbone and node count (LOAD_MATRIX_BACKBONES, default native kafka; LOAD_MATRIX_NODES, default 1 3) and renders them as one table with load/report.js: every scenario's p50/p95/p99, msgs/s, CPU and memory per node, the recovery time after a node kill and the zero-loss checks, one column per backbone and topology. With LOAD_MATRIX_PUBLISH=1 it writes the table to docs/benchmarks/backbone-matrix.md (load/publish_matrix.sh).

# The published numbers: on a quiet machine (nothing else running in Docker), full profile, about 2 hours.
LOAD_MATRIX_PUBLISH=1 load/matrix.sh

# A quick check that the harness works on both backbones (not for publishing):
LOAD_PROFILE=smoke LOAD_SCENARIOS="e2e webhook_burst node_kill" LOAD_MATRIX_NODES=1 load/matrix.sh

The nightly Load workflow runs the same matrix (backbone: [native, kafka] x nodes: [1, 3]), one runner per cell, and its matrix job renders the table into the job summary, uploads it as the backbone-matrix artifact, and on the scheduled run opens a pull request that updates backbone-matrix.md.

In-process write-path benchmark

mix converger.loadtest replaces the former test/load/performance_test.exs. It calls Converger.Activities.create_activity/2 concurrently against the configured database, with no HTTP, WebSocket or deliveries, and prints throughput and p50/p95/p99. Use it to see the cost of the per-conversation seq lock and the insert on a given database. It is not a system benchmark.

mix converger.loadtest --conversations 50 --messages 200 --concurrency 50

It creates a throwaway tenant on an inbound-only channel and deletes it afterwards (--keep keeps it). Run it against a development or scratch database (for example DB_NAME=converger_bench). In the dev environment every query is logged, which slows it down noticeably.

Partition archive and drop at 100M rows

test/load/partition_100m/run.sh checks the retention path of ADR-0034 on a large dataset (#192). It archives the oldest month of about 100M rows to object storage and drops it while writers keep inserting. It runs the production code (Converger.Retention.run/1, Converger.Archive) against its own Postgres and MinIO.

:::note Status Run at 100M rows on 2026-10-15: writers were not blocked, detach took 205 ms and drop 14 ms, and the 16.7M-row month was archived and verified in 530 s. See 2026-10-15, partition archive and drop at 100M rows. :::

What it does:

1. Starts Postgres 17 (the version in docker-compose.yml, tuned for a bulk load: shared_buffers=2GB, max_wal_size=16GB, synchronous_commit left on) and MinIO (the image the CI storage job uses) as the compose project converger-p100m, on 127.0.0.1:15432 and 127.0.0.1:19000, with named volumes. It never touches the development database or another harness.
2. Checks the free space on the Docker disk against an estimate (P100M_BYTES_PER_ROW, about 600 bytes per row for tables and indexes, plus one max_wal_size and the archive, plus 20%). If there is not enough space it stops before seeding (P100M_FORCE=1 overrides).
3. Migrates the schema (MIX_ENV=test mix ecto.migrate). Then it seeds P100M_ROWS rows (60% activities, 40% deliveries) over 6 monthly partitions (2023-07 to 2023-12) and 8 tenants with 2,000 conversations each. It uses INSERT ... SELECT generate_series in batches of 1M rows, and 4 partitions load in parallel. The rows meet the real constraints: the per-conversation seq is unique, every delivery has an activity in the same month, and idempotency_key and provider_message_id are filled for some rows so the partial indexes have entries. Each month is loaded into a plain table and given the per-partition unique indexes and every index of the parent. Then it is vacuumed, analyzed and attached. The seeded partitions have exactly the production indexes, but freshly built, so they are more compact than indexes grown by live traffic.
4. Starts P100M_WRITERS writers (default 8, 25 inserts per second each). They call Converger.Activities.create_activity/2 on the current month and record every latency. After a baseline (P100M_BASELINE_S, 30 s), it runs Converger.Retention.run/1 for the oldest month with the production settings. That means DETACH PARTITION ... CONCURRENTLY, export to MinIO in parts of 50,000 rows, download and SHA-256 check, a row count check, and DROP TABLE. The repo's query telemetry times each step.
5. Downloads every archived object again and counts its lines per tenant and table, and compares the counts with the rows counted in the partition before the detach. It then re-imports one tenant's month with Converger.Archive.import_tenant_month/2, the documented path in Re-importing an archive.
6. Writes results.json, report.md, writers.csv (every insert) and the Postgres log to test/load/partition_100m/.run/out/ (gitignored). The report includes the hardware (host CPU, Docker CPUs and memory, Docker disk, Postgres version and settings) and a linear projection to 100M rows. Then it removes the containers and volumes.

It measures:

• seed time and WAL
• table and index sizes, and bytes per row
• detach, export and verify (per table), and drop durations
• archive object count and size
• WAL generated during the operation (pg_current_wal_lsn() before and after; this includes the writers)
• writer p50, p99 and max before, during the run, during the detach, during the drop, and after
• archive read-back and re-import counts

The test fails if the month is not dropped, if a checksum or row count differs, or if the writers made no progress during the operation.

# the full run: quiet machine, ~90 GB free on the Docker disk
test/load/partition_100m/run.sh

# smoke run (about 2 minutes)
P100M_ROWS=2000000 P100M_BASELINE_S=20 P100M_AFTER_S=10 test/load/partition_100m/run.sh

# keep the seeded database; the next run with the same seed settings reuses it and archives the next month
P100M_KEEP=1 test/load/partition_100m/run.sh

The other settings are listed at the top of the script: months, tenants, conversations, seed batch and parallelism, writers, archive part size, ports and Postgres memory. Run it on a quiet machine. Other containers doing I/O on the same Docker disk show up directly in the writer percentiles (see Publishing a run).

Smoke run, 2M rows, on a developer machine (Ryzen 5 7600X, Docker Desktop on Windows, 12 CPUs, 31 GiB, Postgres 17.11). Another load run shared the machine, so the latencies are only indicative:

	2M rows (smoke)
Seed	20 s (99k rows/s), 1.1 GiB WAL
Size	1.09 GiB tables and indexes, 587 bytes per row
Oldest month	200,000 activities + 133,334 deliveries
DETACH ... CONCURRENTLY (both tables)	63 ms
Export, read back and verify	12.3 s (activities 7.9 s, deliveries 4.2 s)
DROP TABLE (both tables)	26 ms
Archive	16 objects, 28.4 MiB, 89 bytes per row; read back and row counts match
WAL during the run (writers included)	4.4 MiB
Writer p50 / p99 / max, baseline	38 / 119 / 768 ms
Writer p50 / p99 / max, during the run	67 / 227 / 477 ms
Re-import of one tenant (41,666 rows)	6.3 s, counts match

Linear projection to 100M rows: about 87 GiB of disk (55 GiB of data, 16 GiB of WAL, 1.4 GiB of archive, plus 20%), about 17 minutes of seeding, and about 10 minutes to archive and verify the oldest month (16.7M rows). Index builds and I/O do not scale linearly once the data no longer fits in RAM, so expect longer.

Partition drop and archive benchmark

test/load/partition_drop_benchmark_test.exs measures retention of one monthly partition of activities and deliveries while 8 writers insert activities, against Postgres only (excluded from the default suite, tag :benchmark). One test compares the detach and drop with a DELETE of the same rows (ADR-0034); the other (--only retention_archive) runs the whole Converger.Retention.run/1 path with the archive and is the source of Partition retention at scale, which also lists the commands.