Skip to content

hyp purge must not blind queries; LLP 0110 mints the hyp policy verb#319

Merged
philcunliffe merged 2 commits into
masterfrom
fix/purge-query-blindness-llp0110
Jul 16, 2026
Merged

hyp purge must not blind queries; LLP 0110 mints the hyp policy verb#319
philcunliffe merged 2 commits into
masterfrom
fix/purge-query-blindness-llp0110

Conversation

@philcunliffe

@philcunliffe philcunliffe commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

What

Two artifacts from the 2026-07-16 live test of the enrollment privacy flow (LLP 0100-0108, shipped in v1.14.0):

1. Bug fix: hyp purge blinded all queries on a partition.
hyp purge deletes rows by committing puffin position-delete vectors (LLP 0104). Once a snapshot carries deletes, icebird correctly stops exposing a cheap numRows (the live count is unknowable without a scan) and returns undefined. The ai-gateway dataset's skip-empty-partitions guard ((source.numRows ?? 0) > 0) treated that undefined as zero and silently dropped the entire partition from the query union: every query returned empty, with no error or warning. The privacy skill's survey would see an empty cache, and the plan 0108 T3 assertion "purged rows are gone from query results" passed vacuously (everything was gone). The guard now skips only sources known empty (numRows === 0), with an @ref LLP 0104 [constrained-by] on the constraint.

2. LLP 0110 (Accepted issue): retire the hyp ignore --sync misnomer.
The explicit-sync marking shipped spelled hyp ignore --sync <path>, and the LLP 0106 classify-cwd hook prints that inversion into real sessions. LLP 0103 had deliberately left the sync spelling open; 0110 settles the replacement surface as a class-neutral hyp policy set|show|unset|list with the ignore-verb forms kept as deprecated aliases, and LLP 0103 carries the Extended-by: forward-ref on exactly the paragraph that left it open. Implementation is follow-up work driven by the issue.

Verification

  • Reproduced live: post-purge cache returned 0 rows for every query; kernel-level scanRowsFromTable over the same table returned all 113,413 surviving rows, isolating the fault to the dataset guard.
  • With the fix: 124,616 rows visible through the full hyp query sql stack, purged rows still excluded (raw table total 389k), DVs applied correctly.
  • npm test: 2261 tests, 0 fail.
  • ref-check: no errors on the three touched files (the corpus's 101 pre-existing anchor-drift errors are untouched).

Remaining findings from the same test (not in this PR)

  • LLP 0107 skills-ride-attach was never implemented (plan 0108 T7): nothing installs client skills at attach; hyp skills install --client all is the workaround.
  • The backfill client-action ledger survives hyp leave, so a re-enrolling machine skips backfill and the R1 promise "first sync includes your backfilled history" is silently false.
  • Purge leaves all content bytes on disk (old snapshots + DVs) until daemon maintenance compacts; worth confirming maintenance actually reclaims them.

🤖 Generated with Claude Code

Fix: after hyp purge writes puffin position deletes, icebird stops
exposing a cheap numRows (a live count would need a scan), and the
ai-gateway dataset's skip-empty-partitions guard treated that undefined
as zero, silently dropping every purged partition from the union. All
queries on the partition returned empty with no warning, including the
hypaware-privacy skill's survey during the LLP 0100 review window. The
guard now skips only sources known empty (numRows === 0). Found live
testing the enrollment privacy flow on this machine; verified against a
purged-then-backfilled cache (0 rows -> 124,616 visible, purged rows
still excluded).

Docs: LLP 0110 (Accepted issue) records that the explicit-sync marking
shipped as hyp ignore --sync, a semantic inversion the classify-cwd hook
prints into real sessions, and settles the replacement surface: a
class-neutral hyp policy set/show/unset/list, with the ignore-verb forms
kept as deprecated aliases. LLP 0103 carries the Extended-by forward-ref
on exactly the paragraph that left the spelling open.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@philcunliffe philcunliffe added the neutral:adopt Foreign PR adopted into neutral's reconcile scope label Jul 16, 2026
…enrich/gascity dataset unions

The same (source.numRows ?? 0) > 0 guard blinded queries after retention
position-deletes, identically to the ai-gateway fix in this PR (LLP 0104).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@philcunliffe

Copy link
Copy Markdown
Contributor Author

neutral review — round 1 (reviewed head ca66b4b, adopted PR, full-heal)

Verdict: the core fix is correct and a strict improvement. One MAJOR completeness finding was found and fixed by folding in (head moved to 1e56620d).

Core fix (approved): the ai-gateway union guard change (source.numRows ?? 0) > 0 -> source.numRows !== 0 is right. icebird leaves numRows undefined exactly when the current snapshot carries position deletes (a live count would need a scan), so the old guard treated an unknowable count as an empty partition and dropped every partition hyp purge touched. Genuinely-empty partitions still read 0 and are still skipped. Clean style, correct @ref LLP 0104 [constrained-by], well-formed LLP 0110 issue and LLP 0103 Extended-by forward-ref. CI green, MERGEABLE.

MAJOR finding — incomplete fix (now fixed): the identical buggy guard survived in four sibling dataset unions, which the daemon can blind the same way. The root cause is generic: retention writes position deletes for any source-table-layout dataset (src/core/cache/retention.js), and icebird omits numRows under position deletes regardless of whether purge or retention wrote them. After retention deletes some-but-not-all rows from a partition in these datasets, (undefined ?? 0) > 0 is false and the whole partition drops from the query union:

  • hypaware-core/plugins-workspace/otel/src/datasets.js:212
  • hypaware-core/plugins-workspace/context-graph/src/datasets.js:167
  • hypaware-core/plugins-workspace/context-graph-enrich/src/datasets.js:185
  • hypaware-core/plugins-workspace/gascity/src/dataset.js:132

Resolution: folded the identical guard fix (+ retention-aware annotation, same @ref LLP 0104 constraint) into all four in commit 1e56620d. npm test passes (2260). Per the neutral:adopt full-heal contract these were fixed and pushed rather than deferred; revert the commit if you would rather ship the four separately.

Held for you to merge — neutral does not merge adopted PRs. The next tick re-reviews the new head.

@philcunliffe

Copy link
Copy Markdown
Contributor Author

neutral review — round 2 (reviewed head 1e56620d, adopted PR)

Verdict: clean. The only delta since round 1 is the folded-in completeness fix (4 sibling dataset unions: otel, context-graph, context-graph-enrich, gascity). Each applies the identical guard change (source.numRows ?? 0) > 0 -> source.numRows !== 0 with a matching retention-aware annotation and the same @ref LLP 0104 [constrained-by] constraint, mirroring the approved ai-gateway fix exactly. No new findings. CI green (typecheck + test), MERGEABLE.

The PR is now reviewed-clean at the current head. Next tick it reaches the approve verdict and is held for you to merge (neutral does not merge adopted PRs).

@philcunliffe philcunliffe added the neutral:approved neutral reviewed this and holds it for a maintainer merge (own or adopted PR; LLP 0025/0030) label Jul 16, 2026
@philcunliffe

Copy link
Copy Markdown
Contributor Author

neutral verdict: approved (head 1e56620d)

Reviewed clean across two rounds: the ai-gateway query-blindness fix is correct, and the identical guard bug was fixed in all four sibling dataset unions (otel, context-graph, context-graph-enrich, gascity). CI green (typecheck + test), MERGEABLE, mergeable ∧ green ∧ reviewed.

Held for you to merge - neutral does not merge adopted PRs. A push moves the head and re-opens review.

@philcunliffe
philcunliffe merged commit aad4c80 into master Jul 16, 2026
4 checks passed
@philcunliffe
philcunliffe deleted the fix/purge-query-blindness-llp0110 branch July 16, 2026 23:59
@philcunliffe philcunliffe added the neutral:adopted Adoption completion record: merged while carrying neutral:adopt (LLP 0031) label Jul 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

neutral:adopt Foreign PR adopted into neutral's reconcile scope neutral:adopted Adoption completion record: merged while carrying neutral:adopt (LLP 0031) neutral:approved neutral reviewed this and holds it for a maintainer merge (own or adopted PR; LLP 0025/0030)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant