feat(bench): partition-keys benchmark + read-amp repro harness

.gitignore

@@ -10,6 +10,9 @@ clients/ruby/pkg/
 clients/ruby/.bundle/
 clients/ruby/*.gem
 
+# Benchmark harness deps (bun.lock is committed; deps are not)
+benchmark/partition-keys/node_modules/
+
 # Claude Code agent worktrees (ephemeral isolation per agent run)
 .claude/worktrees/
 

benchmark/partition-keys/README.md

@@ -0,0 +1,103 @@
+# Partition-keys read-amplification benchmark
+
+Exercises the pgque v0.8 **partition keys** feature (`sql/pgque-api/partition_keys.sql`)
+at a high-volume multi-tenant scale — the read-amplification scenario in
+`blueprints/partition-keys/SPEC.md` §14 (S4). It uses **only the real installed
+pgque API** (`send` keyed / `subscribe_slot` / `claim_slot` /
+`receive_partitioned` / `ack_partitioned` / `release_slot` and the
+`pgque.partition_slot_status` view) — no demo schema.
+
+## What it measures
+
+The slot mechanism is N independent slot consumers, each scanning the full
+event stream and filtering server-side to its hash class. That gives two
+properties this bench quantifies:
+
+- **R2 — read amplification ~N×.** Each produced event is scanned by all N
+  slots, so buffers touched per produced event scale ~linearly with N. Measured
+  from `pg_stat_statements` (under `track = top`, the event-table scan buffers
+  roll up into the top-level `receive_partitioned` call), at **N=16 vs N=32**.
+- **R7 — a stalled slot pins rotation for the whole queue.** One slot's worker
+  is SIGSTOPped mid-run; its subscription cursor freezes, so the engine cannot
+  drop old `event_N_M` tables. The bench tracks per-slot lag growth, the
+  rotation floor (queue table count), and the catch-up slope after resume.
+
+## Target profile (Fabrizio: >400M events/day)
+
+- **Producer:** 5,000 ev/s sustained keyed `send`, Zipfian tenant skew (s=1.1)
+  over 2,000 tenants, ~200-byte JSON payload — see `producer.sql`. Rate-limited
+  pgbench (`-R 5000 -c 16 -j 8`).
+- **Consumers:** N slot workers (one per slot) on the lease loop — `slot_worker.ts`
+  (bun + node-postgres). Each worker claims its slot, sticky-drains it in
+  batches of 500, acks, releases at drain, re-polls after 200 ms idle. One ack
+  log line per batch: `ts,worker,slot,events,max_ev_id`.
+- **Ticker:** `pk_ticker.py` — `pgque.ticker()` every 250 ms, `pgque.maint()`
+  every 60 s.
+
+## Phases
+
+`run_bench.sh` runs, all output under `/tmp/bench/pk/<phase>/`:
+
+1. **steady-16** (30 min): `bench_q`, consumer `w16`, 16 slots, 5k ev/s.
+2. **steady-32** (30 min): fresh `bench_q32`, consumer `w32`, 32 slots, 5k ev/s.
+3. **stalled-16** (15 min): reuse `bench_q`; slot 7's worker is SIGSTOPped at
+   minute 2 and resumed at minute 10.
+
+Each phase runs, at 5 s / 10 s / 30 s cadences: `slot_status_sampler.sh`
+(per-slot lease + lag from `partition_slot_status`, and queue-level throughput
++ table count from `get_queue_info`), `sys_metrics_sampler.py`,
+`pg_stat_statements_snapshot.py`, and `bloat_sampler.py`. `pg_stat_statements`
+is reset at each phase start and snapshotted at the boundary for the read-amp
+measurement.
+
+Then `summarize.py` parses the CSVs into `summary.md` — producer/consume
+throughput, per-slot pending percentiles, CPU/mem, the N-scaling read-amp
+table, the stalled-slot timeline, and a headline table to paste into a PR.
+
+## Running
+
+On a fresh Hetzner CCX43 (16 dedicated cores, 64 GiB, local NVMe, Ubuntu 24.04),
+Postgres 18 from PGDG:
+
+```bash
+# from the operator machine: ship the repo, then bootstrap
+rsync -a --exclude .git ./ root@VM:/root/pgque/
+ssh root@VM 'bash /root/pgque/benchmark/partition-keys/setup_vm.sh'
+
+# on the VM: install driver deps and run the full ~90 min of measured phases
+ssh root@VM 'cd /root/pgque/benchmark/partition-keys && bun install && \
+  PGUSER=postgres DEVICE=sda bash run_bench.sh'
+```
+
+`DEVICE` is the block device name in `/proc/diskstats` (Hetzner cloud volumes
+usually show as `sda`; adjust if the NVMe is named differently).
+
+### Knobs (env)
+
+`RATE` (5000), `N_SLOTS` (16; steady-32 uses 2×), `DURATION_MIN` (30),
+`STALL_MIN` (15), `STALL_ON_MIN`/`STALL_OFF_MIN`/`STALL_SLOT`, `ROTATION`
+(`30 seconds`), `BATCH` (500), `TTL_S` (30), `PGB_C`/`PGB_J`, `PHASES`
+(`1,2,3`), `OUT` (`/tmp/bench/pk`), and libpq `PGHOST`/`PGUSER`/`PGDATABASE`.
+
+### Smoke run
+
+A 60-second micro-run of phase 1 at 200 ev/s with 4 slots — just env vars:
+
+```bash
+PGHOST=/tmp PGUSER="$(id -un)" PGDATABASE=bench \
+  RATE=200 N_SLOTS=4 DURATION_MIN=1 PHASES=1 DEVICE=none \
+  bash run_bench.sh
+```
+
+## Files
+
+| File | Role |
+|---|---|
+| `setup_vm.sh` | VM bootstrap: PG18 + tuning + bun + pgque install |
+| `producer.sql` | pgbench keyed-send script (`@QUEUE@` rendered at runtime) |
+| `slot_worker.ts` | bun slot-worker lease-loop driver |
+| `package.json` | driver deps (`pg`) |
+| `pk_ticker.py` | ticker + maint loop |
+| `slot_status_sampler.sh` | per-slot lease/lag + queue-rate sampler |
+| `run_bench.sh` | phase orchestrator |
+| `summarize.py` | CSV → markdown report |

benchmark/partition-keys/package.json

@@ -0,0 +1,15 @@
+{
+  "name": "pgque-partition-keys-bench",
+  "private": true,
+  "type": "module",
+  "description": "Slot-worker driver for the partition-keys read-amplification benchmark (bun + node-postgres)",
+  "scripts": {
+    "worker": "bun slot_worker.ts"
+  },
+  "dependencies": {
+    "pg": "^8.16.0"
+  },
+  "devDependencies": {
+    "@types/pg": "^8.15.2"
+  }
+}

benchmark/partition-keys/pk_ticker.py

@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+"""pk_ticker.py -- tight ticker + maintenance loop for the partition-keys bench.
+
+Calls pgque.ticker() every TICK_MS (default 250 ms) so slot cursors advance
+promptly, and pgque.maint() every MAINT_S (default 60 s) so event tables rotate
+and vacuum on the pgque cadence. Persistent autocommit connection, mirroring
+tooling/pgq_ticker_daemon.py.
+
+Env: PGHOST/PGDATABASE/PGUSER (libpq), TICK_MS, MAINT_S, RUN_S (0 = forever).
+"""
+import os
+import signal
+import sys
+import time
+
+import psycopg2
+
+DSN = (
+    f"host={os.environ.get('PGHOST', '127.0.0.1')} "
+    f"dbname={os.environ.get('PGDATABASE', 'bench')} "
+    f"user={os.environ.get('PGUSER', 'postgres')} "
+    "application_name=pk_ticker"
+)
+TICK_S = float(os.environ.get("TICK_MS", "250")) / 1000.0
+MAINT_S = float(os.environ.get("MAINT_S", "60"))
+RUN_S = float(os.environ.get("RUN_S", "0"))
+
+conn = psycopg2.connect(DSN)
+conn.autocommit = True
+cur = conn.cursor()
+
+
+def shutdown(signum, frame):
+    try:
+        conn.close()
+    except Exception:
+        pass
+    sys.exit(0)
+
+
+signal.signal(signal.SIGTERM, shutdown)
+signal.signal(signal.SIGINT, shutdown)
+
+print(f"pk_ticker: tick={TICK_S}s maint={MAINT_S}s run={RUN_S or 'forever'}", flush=True)
+t_start = time.monotonic()
+last_maint = 0.0
+while True:
+    now = time.monotonic()
+    if RUN_S and now - t_start >= RUN_S:
+        break
+    try:
+        cur.execute("select pgque.ticker()")
+        if now - last_maint >= MAINT_S:
+            cur.execute("select pgque.maint()")
+            last_maint = now
+    except Exception as e:  # noqa: BLE001
+        print(f"pk_ticker err: {e}", file=sys.stderr, flush=True)
+        time.sleep(1)
+    time.sleep(TICK_S)
+
+conn.close()

benchmark/partition-keys/producer.sql

@@ -0,0 +1,24 @@
+-- producer.sql -- keyed pgbench producer for the partition-keys read-amp bench.
+--
+-- Zipfian tenant skew (s=1.1) over 2000 tenants models a high-volume multi-tenant
+-- profile: a heavy head of hot buckets plus a long tail. Each send carries a
+-- ~200-byte JSON payload and a partition key of 'tenant-<n>'. The key rides
+-- ev_extra1 (SPEC D1) so every slot's server-side hash filter routes it.
+--
+-- @QUEUE@ is rendered to the live queue name by run_bench.sh (pgbench has no
+-- string-literal variable substitution, so the runner seds this placeholder).
+\set tenant random_zipfian(1, 2000, 1.1)
+select pgque.send(
+    '@QUEUE@',
+    'StorageObjectCreated',
+    json_build_object(
+        'tenant', 'tenant-' || :tenant,
+        'bucket', 'bkt-' || (:tenant % 32),
+        'object', md5(random()::text) || '/' || md5(random()::text) || '.bin',
+        'size_bytes', (random() * 10485760)::bigint,
+        'content_type', 'application/octet-stream',
+        'etag', md5(random()::text),
+        'created_at', clock_timestamp()
+    )::text,
+    'tenant-' || :tenant
+);

benchmark/partition-keys/results/r1-ccx43-summary.md

@@ -0,0 +1,108 @@
+# Partition-keys read-amplification benchmark
+
+High-volume multi-tenant profile: keyed producer + N slot workers on the real
+pgque v0.8 lease API. Maps to SPEC R2 (read amplification ~N x) and R7
+(a stalled slot pins rotation for the whole queue).
+
+## Phase `steady-16`  (N=16, target 5000 ev/s, 30 min)
+
+- Producer: **5,000 ev/s** achieved, 8,996,791 events sent
+- Consumers: **4,999 ev/s** end-to-end, 8,998,023 events acked
+- CPU: 25% avg / 26% peak | RAM used: 3.5 / 5.4 GiB
+- Pending events across slots: p50 0, p99 1,320, max 1,332
+- Read-amp (receive_partitioned): 39,458,647 buffers (39,458,647 hit / 0 read) over 388,732 calls = **4.39 buffers/event**
+- Per-slot consumed: { w0:288,446, w1:303,241, w2:267,539, w3:896,810, w4:499,753, w5:257,939, w6:530,210, w7:327,033, w8:787,695, w9:1,843,421, w10:845,659, w11:862,379, w12:283,429, w13:353,890, w14:298,668, w15:351,911 }
+
+**Bloat & dead tuples**
+
+- Metadata tables by peak dead tuples (end size):
+    - `pgque.subscription`: peak 7,586 dead (end 7,572 dead, 704.00 KiB)
+    - `pgque.tick`: peak 1,608 dead (end 0 dead, 888.00 KiB)
+    - `pgque.partition_slot`: peak 557 dead (end 363 dead, 96.00 KiB)
+    - `pgque.partition_consumer`: peak 35 dead (end 35 dead, 32.00 KiB)
+    - `pgque.consumer`: peak 20 dead (end 20 dead, 48.00 KiB)
+- `pgque.partition_slot` (v0.8 lease, 2 UPDATEs/batch): peak **557** dead, end 363 dead, 96.00 KiB — HOT updates should keep this tiny
+- `pgque.tick` peak dead tuples: 1,608 (rotation-metadata churn)
+- `pgque.subscription` peak dead tuples: 7,586 (rotation-metadata churn)
+- Event tables: 4 present at phase start -> 4 at end (rotation dropping old `event_N_M`)
+- Event-table bytes: start 56.00 KiB, end 3.57 GiB, peak 3.57 GiB
+- Event-table peak dead tuples: 0 (append-only — expect ~0; nonzero means retries)
+- Total DB footprint: end 3.57 GiB, peak 3.57 GiB
+
+## Phase `steady-32`  (N=32, target 5000 ev/s, 30 min)
+
+- Producer: **5,001 ev/s** achieved, 8,997,460 events sent
+- Consumers: **5,000 ev/s** end-to-end, 9,000,840 events acked
+- CPU: 30% avg / 31% peak | RAM used: 7.3 / 9.3 GiB
+- Pending events across slots: p50 0, p99 1,318, max 1,337
+- Read-amp (receive_partitioned): 63,166,134 buffers (63,166,134 hit / 0 read) over 775,996 calls = **7.02 buffers/event**
+- Per-slot consumed: { w0:145,553, w1:142,039, w2:185,695, w3:451,545, w4:95,106, w5:116,060, w6:196,430, w7:166,868, w8:331,859, w9:1,707,562, w10:118,269, w11:782,539, w12:121,190, w13:219,230, w14:122,795, w15:211,793, w16:143,608, w17:160,798, w18:82,634, w19:447,463, w20:403,448, w21:141,426, w22:335,823, w23:159,336, w24:453,227, w25:136,178, w26:728,446, w27:80,067, w28:161,406, w29:135,008, w30:176,196, w31:141,243 }
+
+**Bloat & dead tuples**
+
+- Metadata tables by peak dead tuples (end size):
+    - `pgque.subscription`: peak 9,502 dead (end 1,878 dead, 1.26 MiB)
+    - `pgque.partition_slot`: peak 704 dead (end 578 dead, 96.00 KiB)
+    - `pgque.partition_consumer`: peak 66 dead (end 0 dead, 64.00 KiB)
+    - `pgque.consumer`: peak 20 dead (end 20 dead, 48.00 KiB)
+    - `pgque.queue`: peak 8 dead (end 8 dead, 48.00 KiB)
+- `pgque.partition_slot` (v0.8 lease, 2 UPDATEs/batch): peak **704** dead, end 578 dead, 96.00 KiB — HOT updates should keep this tiny
+- `pgque.tick` peak dead tuples: 0 (rotation-metadata churn)
+- `pgque.subscription` peak dead tuples: 9,502 (rotation-metadata churn)
+- Event tables: 8 present at phase start -> 8 at end (rotation dropping old `event_N_M`)
+- Event-table bytes: start 3.62 GiB, end 7.19 GiB, peak 7.19 GiB
+- Event-table peak dead tuples: 0 (append-only — expect ~0; nonzero means retries)
+- Total DB footprint: end 7.19 GiB, peak 7.19 GiB
+
+## Phase `stalled-16`  (N=16, target 5000 ev/s, 15 min)
+
+- Producer: **4,997 ev/s** achieved, 4,495,477 events sent
+- Consumers: **4,996 ev/s** end-to-end, 4,496,137 events acked
+- CPU: 25% avg / 30% peak | RAM used: 8.9 / 9.0 GiB
+- Pending events across slots: p50 0, p99 1,665,413, max 2,386,485
+- Read-amp (receive_partitioned): 13,361,585 buffers (13,361,585 hit / 0 read) over 189,979 calls = **2.97 buffers/event**
+- Per-slot consumed: { w0:143,742, w1:150,971, w2:133,571, w3:449,575, w4:249,337, w5:128,238, w6:265,385, w7:163,156, w8:392,808, w9:921,401, w10:423,363, w11:431,755, w12:141,882, w13:176,922, w14:148,272, w15:175,759 }
+
+**Bloat & dead tuples**
+
+- Metadata tables by peak dead tuples (end size):
+    - `pgque.subscription`: peak 8,912 dead (end 1,672 dead, 1.05 MiB)
+    - `pgque.tick`: peak 7,205 dead (end 0 dead, 1.70 MiB)
+    - `pgque.partition_slot`: peak 640 dead (end 329 dead, 96.00 KiB)
+    - `pgque.consumer`: peak 36 dead (end 36 dead, 48.00 KiB)
+    - `pgque.partition_consumer`: peak 16 dead (end 16 dead, 64.00 KiB)
+- `pgque.partition_slot` (v0.8 lease, 2 UPDATEs/batch): peak **640** dead, end 329 dead, 96.00 KiB — HOT updates should keep this tiny
+- `pgque.tick` peak dead tuples: 7,205 (rotation-metadata churn)
+- `pgque.subscription` peak dead tuples: 8,912 (rotation-metadata churn)
+- Event tables: 8 present at phase start -> 8 at end (rotation dropping old `event_N_M`)
+- Event-table bytes: start 3.62 GiB, end 5.38 GiB, peak 5.38 GiB
+- Event-table peak dead tuples: 0 (append-only — expect ~0; nonzero means retries)
+- Total DB footprint: end 5.38 GiB, peak 5.38 GiB
+- R7 pin/release — event-table bytes: stall start 3.62 GiB -> stall end 5.38 GiB -> phase end 5.38 GiB (grows while the stalled cursor pins rotation, drops after resume)
+
+## Read amplification: N-scaling (SPEC R2)
+
+| N | buffers/event | read/event | ratio vs smallest N |
+|--:|--:|--:|--:|
+| 16 | 4.39 | 0.00 | 1.00x |
+| 32 | 7.02 | 0.00 | 1.60x |
+
+Every slot scans the full stream and filters server-side, so buffers touched per produced event scale ~linearly with N. Observed 32/16 = 1.60x (ideal 2x).
+
+## Stalled-slot: rotation pinning (SPEC R7)
+
+Stalled slot (by peak lag): **slot 7**.
+- Lease-expiry stall window: 896s (no live owner)
+- Pending events: baseline 0 -> peak 2,386,485
+- Lag growth during stall: **2,663 events/s**
+- Catch-up after resume: did not return to baseline within the phase
+- Rotation floor: queue held 3..3 event tables (the stalled slot pins the drop floor)
+
+## Headline
+
+| Phase | N | Producer ev/s | Consume ev/s | Pending p99 | Buffers/event | CPU peak | Peak dead tup (meta) | Event tbl GiB end |
+|---|--:|--:|--:|--:|--:|--:|--:|--:|
+| steady-16 | 16 | 5,000 | 4,999 | 1,320 | 4.39 | 26% | 7,586 | 3.57 |
+| steady-32 | 32 | 5,001 | 5,000 | 1,318 | 7.02 | 31% | 9,502 | 7.19 |
+| stalled-16 | 16 | 4,997 | 4,996 | 1,665,413 | 2.97 | 30% | 8,912 | 5.38 |
+