2026-10-15: partition archive and drop at 100M rows

This run measures the exit criterion of #59 that was tracked in #192: partition drop and archive verified on a 100M-row dataset. The command was test/load/partition_100m/run.sh with its defaults (how it works).

Findings

• Verified at 100M rows. 100 013 776 rows (60M activities, 40M deliveries, 6 months, 8 tenants; 54.6 GiB). Converger.Retention.run/1 handled the oldest month (16 666 667 rows), with the production code and settings: it detached the partitions concurrently, exported them to MinIO, read the export back, verified it and dropped them. The 336 archive objects (1.45 GiB) were downloaded again, and their SHA-256 hashes and per-tenant row counts matched the partition counted before the detach. One tenant re-imported with matching counts.
• Writers were not blocked. 8 writers inserted 25 rows/s each through Activities.create_activity/2 the whole time, with 0 failed inserts. Their p99 was 144 ms in the baseline and 124 ms during the run. During the detach it was 186 ms, and during the drop 129 ms; the maximum over the whole run was 309 ms.
• Detach and drop stay constant. DETACH PARTITION ... CONCURRENTLY took 205 ms and DROP TABLE 14 ms for both tables. Export and verification are what grow with the month: 530 s for 16.7M rows, about 31 000 rows/s. That is slower than the 57 000–59 000 rows/s at 10M–30M rows in #196's run, because this month did not fit in Postgres' 2 GiB of shared buffers.
• Conditions: a developer machine (Ryzen 5 7600X, Docker Desktop on Windows, 12 CPUs, 31 GiB for Docker). During the seed, a load test from another worktree ran in Docker at the same time, so the seed time is pessimistic. The retention-phase numbers are within the range of the quiet smoke runs.

Report

Generated 2026-10-15 03:09:33Z by test/load/partition_100m/run.sh.

Hardware

Host	MINGW64_NT-10.0-26200 3.6.5-22c95533.x86_64 x86_64, 62.88 GiB memory
CPU	AMD Ryzen 5 7600X 6-Core Processor
Docker	Docker Desktop, 12 CPUs, 30.79 GiB memory, server 29.8.2
Docker disk	1006.85 GiB total, 930.28 GiB free before the run
Postgres	17.11, shared_buffers 2GB, max_wal_size 16GB, synchronous_commit on

Dataset

60000000 activities + 40000000 deliveries, 6 months, 8 tenants. Seed: 937.0 s (106729 rows/s, 4 jobs), WAL 54.43 GiB.

Table	Heap	Indexes	Total
activities	20.26 GiB	11.86 GiB	32.12 GiB
deliveries	6.92 GiB	15.55 GiB	22.47 GiB

Database 54.62 GiB; 586.1 bytes per row (tables + indexes).

Oldest month 2023-07

10000000 activities + 6666667 deliveries, archived by Converger.Retention.run/1 to MinIO.

Step	Result
DETACH PARTITION ... CONCURRENTLY (both tables)	205.2 ms
Export + read back + verify	529.7 s (activities 395.3 s, deliveries 133.9 s)
DROP TABLE (both tables)	14.1 ms
Whole run	529.9 s
WAL during the run (writers included)	319.8 MiB
Archive	336 objects, 1.45 GiB (93.1 bytes per row), 16666667 rows, all verified: true
Partition gone afterwards	true

Writers (8 x 25/s, Activities.create_activity/2)

Window	Inserts	p50 ms	p99 ms	max ms
before (baseline)	4854	36.97	144.18	187.7
during the whole run	97019	27.95	123.9	308.84
during the detach	71	58.27	185.75	185.75
during the drop	24381	28.16	128.51	174.18
after	2921	22.63	32.77	43.72

Failed inserts: 0.

Read back

336 objects (1.45 GiB) downloaded in 10.4 s: SHA-256 and line counts match the manifest; per tenant and table, archived rows equal the rows counted in the partition before the detach (16666667 rows).

Re-import (Converger.Archive.import_tenant_month/2):

Tenant	Rows in DB (activities / deliveries)	Time	Counts match
12b7c223-2efd-4caf-b9e9-47f8250bd84c	1250000 / 833334	210.5 s (9896 rows/s)	yes

Projection to 100000000 rows

Disk needed about 86.43 GiB (54.58 GiB data, one max_wal_size, 1.45 GiB archive, +20%); seed about 15.6 min; archive and verify of the oldest month (16664371 rows) about 8.8 min. Linear extrapolation from this run; index builds and I/O do not scale linearly once data exceeds RAM.