Skip to main content

Data retention, partitions and archive

activities and deliveries grow with traffic, so they are stored in monthly partitions and old months are archived to object storage and removed according to each tenant's retention. Smaller operational tables (channel_health_checks, audit_logs) are pruned by age. The design and the options that were rejected are in ADR-0034; the migration that converts an existing installation is described in Migrations.

Monthly partitions​

TablePartition keyPartition name
activitiesinserted_atactivities_pYYYY_MM
deliveriesactivity_inserted_at (the inserted_at of the delivery's activity)deliveries_pYYYY_MM

A partition covers one calendar month in UTC, [YYYY-MM-01 00:00, first day of next month). A delivery always lives in the same month as its activity, even when it is created weeks later by a retry, so a month of activities and its deliveries are archived and dropped together.

Partitions are created ahead of time: the current month and the next PARTITION_MONTHS_AHEAD months (default 3). This runs in the migration, once when the application boots, and daily at 00:15 UTC (Converger.Workers.PartitionMaintenanceWorker). There is no DEFAULT partition (it would prevent the non-blocking detach), so an insert into a month without a partition fails: keep at least one future month in place. To check:

SELECT inhrelid::regclass FROM pg_inherits
WHERE inhparent = 'activities'::regclass ORDER BY 1 DESC LIMIT 3;

Unique indexes that cannot contain the partition key exist on every partition rather than on the parent: (conversation_id, seq) and (conversation_id, idempotency_key). The application keeps them unique across partitions (the seq counter on conversations, and an idempotency re-check under the conversation lock); see the ADR. The partitioned tables have no foreign keys: deleting a tenant, channel or conversation enqueues a Converger.Workers.PurgeWorker job in the same transaction that deletes their activities and deliveries in batches of 5,000 (see Tenants).

Per-tenant retention​

Every tenant has retention_days (default 365, editable in the admin tenant form; never below RETENTION_MIN_DAYS, default 30). A row is expired once the whole month it belongs to is older than the tenant's retention.

Converger.Workers.RetentionWorker runs on the 1st of every month at 02:00 UTC on the maintenance queue. For every month that ended at least RETENTION_MIN_DAYS ago:

  • Whole month (the common case, all tenants on the same retention): the partitions are detached with DETACH PARTITION ... CONCURRENTLY, which waits for transactions that started before it but never blocks reads or inserts, archived from the detached tables (which nobody writes any more), verified, and dropped.
  • Some tenants expired (a tenant with a shorter retention than others in the same month): only that tenant's rows are archived and deleted from the live partitions, one verified part per transaction. The partition is dropped when the last tenant in it expires. A tenant with a much longer retention than everyone else keeps those months' partitions alive, and the others' rows are then deleted row by row: still correct, but slower.
  • Rows of tenants that no longer exist count as expired (they are archived too, not silently dropped).

Nothing is dropped or deleted unless its archive object exists with the recorded size and SHA-256 and the archived row count equals the partition's row count. On any failure (storage outage, checksum mismatch) the run stops with an error, Oban retries it with backoff (up to about a day between attempts, 10 attempts) and the next run continues where it stopped: a detached month whose archive has not finished is processed first, and parts already uploaded and recorded are not exported again. While a month is detached its rows are not visible to the API, exactly as after the drop, but they are still in the database.

How long it takes. Measured on a developer machine with 8 writers inserting the whole time (Partition retention at scale): a month of 30M activities and 30M deliveries (23 GB of partitions) was detached in 24 ms, archived and verified in 17 minutes (about 59,000 rows/s, 2.8 GB of archive) and dropped in 6 s, the time Postgres needs to unlink the files at commit. Detach does not grow with the month; the archive grows linearly with the rows (a 100M-row month: roughly an hour, extrapolated). No step blocked the writers. The worker has no job timeout, so a long run is not cut off; plan the archive storage at about 12 % of the partition size (gzipped JSON Lines of the benchmark's narrow rows; wider rows compress differently).

Run it now instead of waiting for the 1st (on a running node; the job is unique, so it never runs twice at once):

bin/converger rpc "Converger.Release.run_retention()"

Inspect what has been archived:

SELECT table_name, month, mode, count(*) AS parts, sum(row_count) AS rows, max(verified_at)
FROM archive_parts GROUP BY 1, 2, 3 ORDER BY 2, 1;

Retention archive​

Archived rows are gzip-compressed JSON Lines: one row per line with every column, as Postgres row_to_json writes it (timestamps are UTC without an offset, like the columns).

archive/<tenant_id>/<YYYY-MM>/activities-00001.jsonl.gz
archive/<tenant_id>/<YYYY-MM>/activities-00002.jsonl.gz
archive/<tenant_id>/<YYYY-MM>/deliveries-00001.jsonl.gz

Each part holds at most ARCHIVE_PART_ROWS rows (default 50,000) in primary-key order; part numbers are consecutive from 1. Every part is recorded in archive_parts with its object key, row count, size, SHA-256 and mode (detached for whole months, deleted for per-tenant deletes). The archive is written through the attachment storage backend (File storage): local disk, S3, MinIO, R2, GCS or Azure.

Protect the archive like a backup: enable bucket versioning or object lock, restrict delete permissions, and set the bucket's own lifecycle rules (for example a move to a colder storage class) as needed. Converger never deletes archive objects.

Re-importing an archive​

# every part of one tenant and month (activities, then deliveries)
mix converger.archive.import --tenant 6f1c2a0e-... --month 2025-01
# one object, or a local file downloaded from the bucket
mix converger.archive.import --key archive/6f1c2a0e-.../2025-01/activities-00001.jsonl.gz
mix converger.archive.import --file ./activities-00001.jsonl.gz

# in a release (no Mix)
bin/converger eval 'Converger.Release.import_archive(tenant: "6f1c2a0e-...", month: "2025-01")'

The importer creates the month's partitions if they were dropped, checks objects listed in archive_parts against their SHA-256, and inserts with ON CONFLICT DO NOTHING, so an import can be repeated. It reads parts 1, 2, ... until one is missing, so it also works against a database restored without archive_parts. Re-imported rows are subject to retention again on the next run: raise the tenant's retention_days first if they should stay, or import into a separate database.

Checking it at scale​

test/load/partition_100m/run.sh seeds about 100M rows into its own Postgres. While writers keep inserting, it runs this retention path for the oldest month against MinIO. It measures detach, export and verify, drop, the writers' p99 and the WAL, then reads the archive back and re-imports a tenant to compare row counts. How to run it, the disk it needs and a smoke-run result are in Benchmarks. At 100M rows the writers were not blocked, detach took 205 ms and drop 14 ms, and archiving the 16.7M-row month took 530 s (2026-10-15 run, #192).

Pruning other tables​

Converger.Workers.PruneWorker runs daily at 01:30 UTC and deletes, 10,000 rows per statement:

TableWindowVariable
channel_health_checks7 days (by checked_at)HEALTH_CHECK_RETENTION_DAYS
audit_logs365 days (by inserted_at)AUDIT_LOG_RETENTION_DAYS
webhook_events (platform event webhooks)7 days (by inserted_at)WEBHOOK_EVENT_RETENTION_DAYS
backbone_receipts (event backbone idempotency receipts)7 days (by inserted_at)BACKBONE_RECEIPT_RETENTION_DAYS

0 disables a window. These rows are not archived; export them first if you need them longer (for example COPY (SELECT * FROM audit_logs WHERE inserted_at < ...) TO ...).

Configuration​

VariableDefaultEffect
RETENTION_MIN_DAYS30Platform minimum for tenants.retention_days; months younger than this are never touched.
HEALTH_CHECK_RETENTION_DAYS7Health check pruning window, 0 disables.
AUDIT_LOG_RETENTION_DAYS365Audit log pruning window, 0 disables.
WEBHOOK_EVENT_RETENTION_DAYS7Platform webhook event log pruning window, 0 disables.
BACKBONE_RECEIPT_RETENTION_DAYS7Event backbone idempotency receipts. Keep it longer than any redelivery or dual-publish window (an event older than its receipt would be applied again); 0 disables pruning.
PARTITION_MONTHS_AHEAD3Months of future partitions kept in place.
ARCHIVE_BUCKET / ARCHIVE_CONTAINER / ARCHIVE_DIRunsetArchive to a different bucket (S3, MinIO, R2, GCS), Azure container or local directory, with the attachment storage credentials. Unset: the attachment bucket.
ARCHIVE_PREFIXarchiveFirst path segment of archive keys.
ARCHIVE_PART_ROWS50000Rows per archive object.

The application keys (config :converger, Converger.Retention, Converger.Archive, Converger.Partitions) are listed in Configuration. The cron schedule is in config/config.exs (Oban.Plugins.Cron).

Monitoring​

  • Oban jobs: a Converger.Workers.RetentionWorker in retryable or discarded (Oban dashboard, /admin/oban) means a month could not be archived; its error says why (upload_failed, checksum_mismatch, archive_incomplete).

  • Detached months waiting for their archive:

    SELECT relname FROM pg_class c
    WHERE relkind = 'r' AND relname ~ '^(activities|deliveries)_p[0-9]{4}_[0-9]{2}$'
      AND NOT EXISTS (SELECT 1 FROM pg_inherits i WHERE i.inhrelid = c.oid);

  • Telemetry: [:converger, :archive, :part] with rows and bytes per uploaded part.

To make a detached month visible again instead of archiving it (for example after raising a tenant's retention), re-attach it: ALTER TABLE activities ATTACH PARTITION activities_p2025_01 FOR VALUES FROM ('2025-01-01') TO ('2025-02-01'); (same for deliveries).