Skip to content

feat(improvement): source both proposers uniformly from agent-eval/campaign #365

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
drewstone merged 1 commit into from
Jun 22, 2026
Merged

feat(improvement): source both proposers uniformly from agent-eval/campaign #365

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 18 additions & 18 deletions docs/api/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2648,7 +2648,7 @@ Test seam — inject the worktree-dirty check (defaults to `git status`).

### ImproveOptions

Defined in: [improvement/improve.ts:48](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L48)
Defined in: [improvement/improve.ts:47](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L47)

#### Type Parameters

Expand All @@ -2666,7 +2666,7 @@ Defined in: [improvement/improve.ts:48](https://github.com/tangle-network/agent-

> `optional` **surface?**: [`ImproveSurface`](#improvesurface)

Defined in: [improvement/improve.ts:51](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L51)
Defined in: [improvement/improve.ts:50](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L50)

Which profile lever to optimize. Default `'prompt'`. Selects the default
generator + the baseline-surface extraction shape.
Expand All @@ -2675,7 +2675,7 @@ Which profile lever to optimize. Default `'prompt'`. Selects the default

> `optional` **generator?**: `SurfaceProposer`\<`unknown`\>

Defined in: [improvement/improve.ts:55](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L55)
Defined in: [improvement/improve.ts:54](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L54)

The `SurfaceProposer` that mutates the surface. When unset, the facade
picks the default for `surface` (`gepaProposer` for prompt, `skillOptProposer`
Expand All @@ -2685,7 +2685,7 @@ The `SurfaceProposer` that mutates the surface. When unset, the facade

> `optional` **gate?**: `"none"` \| `"holdout"`

Defined in: [improvement/improve.ts:58](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L58)
Defined in: [improvement/improve.ts:57](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L57)

Gate mode. `'holdout'` (default) runs the held-out promotion gate;
`'none'` is a baseline-only run (`budget.generations = 0`).
Expand All @@ -2694,23 +2694,23 @@ Gate mode. `'holdout'` (default) runs the held-out promotion gate;

> **scenarios**: `TScenario`[]

Defined in: [improvement/improve.ts:60](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L60)
Defined in: [improvement/improve.ts:59](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L59)

Scenarios to evaluate against. Passthrough to `selfImprove`.

##### judge

> **judge**: `JudgeConfig`\<`TArtifact`, `TScenario`\>

Defined in: [improvement/improve.ts:62](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L62)
Defined in: [improvement/improve.ts:61](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L61)

Judge that scores artifacts. Passthrough to `selfImprove`.

##### agent

> **agent**: (`surface`, `scenario`, `ctx`) => `Promise`\<`TArtifact`\>

Defined in: [improvement/improve.ts:65](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L65)
Defined in: [improvement/improve.ts:64](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L64)

The agent under improvement — same shape as `selfImprove.agent`: it takes
the current surface + scenario + ctx and returns the artifact to judge.
Expand All @@ -2737,15 +2737,15 @@ The agent under improvement — same shape as `selfImprove.agent`: it takes

> `optional` **budget?**: `SelfImproveBudget`

Defined in: [improvement/improve.ts:67](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L67)
Defined in: [improvement/improve.ts:66](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L66)

Budget + loop shape. Passthrough; `gate: 'none'` forces `generations = 0`.

##### llm?

> `optional` **llm?**: `SelfImproveLlm`

Defined in: [improvement/improve.ts:70](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L70)
Defined in: [improvement/improve.ts:69](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L69)

LLM config. Passthrough to `selfImprove` AND used to construct the default
reflective proposer (`gepaProposer`/`skillOptProposer`) when `generator` is unset.
Expand All @@ -2754,7 +2754,7 @@ LLM config. Passthrough to `selfImprove` AND used to construct the default

> `optional` **allowedModels?**: readonly `string`[]

Defined in: [improvement/improve.ts:74](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L74)
Defined in: [improvement/improve.ts:73](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L73)

Restrict the run to this subset of models. When set, the reflection model
(`llm.model`, or the default when unset) must be a member, or `improve()` throws
Expand All @@ -2764,7 +2764,7 @@ Restrict the run to this subset of models. When set, the reflection model

### ImproveResult

Defined in: [improvement/improve.ts:77](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L77)
Defined in: [improvement/improve.ts:76](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L76)

#### Type Parameters

Expand All @@ -2782,7 +2782,7 @@ Defined in: [improvement/improve.ts:77](https://github.com/tangle-network/agent-

> **profile**: `AgentProfile`

Defined in: [improvement/improve.ts:80](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L80)
Defined in: [improvement/improve.ts:79](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L79)

The profile after improvement: the winner surface applied back into the
matching field when the gate shipped, else the input profile unchanged.
Expand All @@ -2791,31 +2791,31 @@ The profile after improvement: the winner surface applied back into the

> **shipped**: `boolean`

Defined in: [improvement/improve.ts:82](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L82)
Defined in: [improvement/improve.ts:81](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L81)

True when `gateDecision === 'ship'`.

##### lift

> **lift**: `number`

Defined in: [improvement/improve.ts:84](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L84)
Defined in: [improvement/improve.ts:83](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L83)

Held-out lift (`winner − baseline` composite).

##### gateDecision

> **gateDecision**: `"ship"` \| `"hold"` \| `"need_more_work"` \| `"model_ceiling"` \| `"arch_ceiling"`

Defined in: [improvement/improve.ts:86](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L86)
Defined in: [improvement/improve.ts:85](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L85)

The five-valued gate verdict from `selfImprove`.

##### raw

> **raw**: `SelfImproveResult`\<`TScenario`, `TArtifact`\>

Defined in: [improvement/improve.ts:88](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L88)
Defined in: [improvement/improve.ts:87](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L87)

Full `selfImprove` result for advanced inspection.

Expand Down Expand Up @@ -6380,7 +6380,7 @@ Verifies the edited worktree. Sync or async; throws only on a setup fault

> **ImproveSurface** = `"prompt"` \| `"skills"` \| `"tools"` \| `"mcp"` \| `"hooks"` \| `"code"`

Defined in: [improvement/improve.ts:46](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L46)
Defined in: [improvement/improve.ts:45](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L45)

The agent-profile lever `improve` optimizes. Mirrors the AgentProfile-law
profile levers; `code` is the implementation-tier surface.
Expand Down Expand Up @@ -7774,7 +7774,7 @@ Defined in: [improvement/build-prompts.ts:43](https://github.com/tangle-network/

> **improve**\<`TScenario`, `TArtifact`\>(`profile`, `findings`, `opts`): `Promise`\<[`ImproveResult`](#improveresult)\<`TScenario`, `TArtifact`\>\>

Defined in: [improvement/improve.ts:200](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L200)
Defined in: [improvement/improve.ts:199](https://github.com/tangle-network/agent-runtime/blob/main/src/improvement/improve.ts#L199)

Run the held-out-gated self-improvement loop on ONE profile surface.

Expand Down
28 changes: 14 additions & 14 deletions docs/api/lifecycle.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1376,7 +1376,7 @@ The shared baseline eval (the "without" arm, measured once).

### SkillDraft

Defined in: [lifecycle/skill-generator.ts:32](https://github.com/tangle-network/agent-runtime/blob/main/src/lifecycle/skill-generator.ts#L32)
Defined in: [lifecycle/skill-generator.ts:33](https://github.com/tangle-network/agent-runtime/blob/main/src/lifecycle/skill-generator.ts#L33)

A distilled skill draft: a name + the `SKILL.md` body.

Expand All @@ -1386,47 +1386,47 @@ A distilled skill draft: a name + the `SKILL.md` body.

> **name**: `string`

Defined in: [lifecycle/skill-generator.ts:34](https://github.com/tangle-network/agent-runtime/blob/main/src/lifecycle/skill-generator.ts#L34)
Defined in: [lifecycle/skill-generator.ts:35](https://github.com/tangle-network/agent-runtime/blob/main/src/lifecycle/skill-generator.ts#L35)

Skill name — becomes the inline resource ref name + the artifact name.

##### content

> **content**: `string`

Defined in: [lifecycle/skill-generator.ts:36](https://github.com/tangle-network/agent-runtime/blob/main/src/lifecycle/skill-generator.ts#L36)
Defined in: [lifecycle/skill-generator.ts:37](https://github.com/tangle-network/agent-runtime/blob/main/src/lifecycle/skill-generator.ts#L37)

The `SKILL.md` document body (markdown).

##### description?

> `optional` **description?**: `string`

Defined in: [lifecycle/skill-generator.ts:38](https://github.com/tangle-network/agent-runtime/blob/main/src/lifecycle/skill-generator.ts#L38)
Defined in: [lifecycle/skill-generator.ts:39](https://github.com/tangle-network/agent-runtime/blob/main/src/lifecycle/skill-generator.ts#L39)

Optional one-line description for review surfaces.

***

### SkillGeneratorOptions

Defined in: [lifecycle/skill-generator.ts:56](https://github.com/tangle-network/agent-runtime/blob/main/src/lifecycle/skill-generator.ts#L56)
Defined in: [lifecycle/skill-generator.ts:57](https://github.com/tangle-network/agent-runtime/blob/main/src/lifecycle/skill-generator.ts#L57)

#### Properties

##### distill

> **distill**: [`DistillSkills`](#distillskills)

Defined in: [lifecycle/skill-generator.ts:58](https://github.com/tangle-network/agent-runtime/blob/main/src/lifecycle/skill-generator.ts#L58)
Defined in: [lifecycle/skill-generator.ts:59](https://github.com/tangle-network/agent-runtime/blob/main/src/lifecycle/skill-generator.ts#L59)

REQUIRED — the create step. Without it there is no skill to optimize.

##### refine?

> `optional` **refine?**: [`RefineSkill`](#refineskill)

Defined in: [lifecycle/skill-generator.ts:60](https://github.com/tangle-network/agent-runtime/blob/main/src/lifecycle/skill-generator.ts#L60)
Defined in: [lifecycle/skill-generator.ts:61](https://github.com/tangle-network/agent-runtime/blob/main/src/lifecycle/skill-generator.ts#L61)

OPTIONAL — the optimize step. Omit to ship distilled drafts unrefined.

Expand Down Expand Up @@ -1917,7 +1917,7 @@ test injects a pure function. Returns up to `count` drafts.

> **DistillSkills** = (`ctx`) => `Promise`\<[`SkillDraft`](#skilldraft)[]\> \| [`SkillDraft`](#skilldraft)[]

Defined in: [lifecycle/skill-generator.ts:47](https://github.com/tangle-network/agent-runtime/blob/main/src/lifecycle/skill-generator.ts#L47)
Defined in: [lifecycle/skill-generator.ts:48](https://github.com/tangle-network/agent-runtime/blob/main/src/lifecycle/skill-generator.ts#L48)

DISTILL — create new skill drafts from the agent's history. Returns zero or
more drafts (zero is valid: nothing worth distilling this round). The
Expand All @@ -1940,11 +1940,11 @@ LLM; a test injects a pure function.

> **RefineSkill** = (`draft`) => `Promise`\<[`SkillDraft`](#skilldraft)\> \| [`SkillDraft`](#skilldraft)

Defined in: [lifecycle/skill-generator.ts:54](https://github.com/tangle-network/agent-runtime/blob/main/src/lifecycle/skill-generator.ts#L54)
Defined in: [lifecycle/skill-generator.ts:55](https://github.com/tangle-network/agent-runtime/blob/main/src/lifecycle/skill-generator.ts#L55)

REFINE — improve ONE distilled draft (wording, structure, examples). The
production implementation wraps `runSkillOpt`. Returns the refined draft; when
omitted from `skillGenerator`, the distilled draft is used as-is.
production implementation drives `skillOptProposer`. Returns the refined draft;
when omitted from `skillGenerator`, the distilled draft is used as-is.

#### Parameters

Expand Down Expand Up @@ -2540,7 +2540,7 @@ Cold start on a fixture domain (the closed loop in one call):

> **skillGenerator**(`opts`): [`CandidateGenerator`](#candidategenerator)\<`"skill"`\>

Defined in: [lifecycle/skill-generator.ts:74](https://github.com/tangle-network/agent-runtime/blob/main/src/lifecycle/skill-generator.ts#L74)
Defined in: [lifecycle/skill-generator.ts:75](https://github.com/tangle-network/agent-runtime/blob/main/src/lifecycle/skill-generator.ts#L75)

Build a `CandidateGenerator` for the skill surface that distills new skills
from history, then (optionally) refines them, and emits each as a `skill`
Expand All @@ -2559,10 +2559,10 @@ artifact carrying an inline `SKILL.md` resource ref.
#### Example

```ts
Production wiring (distill = LLM reflection, refine = skillOpt):
Production wiring (distill = LLM reflection, refine = skillOptProposer):
skillGenerator({
distill: reflectiveDistill, // creates the draft from traces
refine: skillOptRefine, // optimizes the draft
refine: skillOptRefine, // optimizes the draft via skillOptProposer
})
```

Expand Down
4 changes: 2 additions & 2 deletions docs/canonical-api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

<!-- This doc is the JUDGMENT layer: the mental model (§1), the AgentProfile law (§1.5), and the anti-reinvention decision table (§2). Per-symbol signatures + `file:line` are GENERATED into `docs/api/` (TypeDoc, do NOT hand-edit) — that is the mechanical reference. The freshness gate (`pnpm docs:freshness`) FAILS CI if a version pin, a cited `file:line`, or a decision-table symbol drifts from source — see `docs/MAINTAINING.md`. Keep this file the small, hand-curated spine. -->

> **Version 0.74.0.** Per-symbol signatures live in the generated `docs/api/` reference (one page per module). The pinned substrate is agent-eval `>=0.95.0 <1.0.0`; the sandbox substrate that materializes profiles into harness shapes is `@tangle-network/sandbox` (peer `>=0.8.0 <1.0.0`). The neutral contract types (`AgentProfile`, `AgentProfileMcpServer`, `HarnessType`, `ReasoningEffort`, `Part`/`ToolPart`/`ToolState`) are owned by **`@tangle-network/agent-interface`** (peer `>=0.10.0 <1.0.0`) — the single source of truth. Substrate symbols (`selfImprove`/`gepaProposer`/`defaultProductionGate`/`heldOutGate`/`pairedBootstrap`/…) are re-exported through `@tangle-network/agent-eval/contract` (or `/campaign`), not local to this package.
> **Version 0.75.0.** Per-symbol signatures live in the generated `docs/api/` reference (one page per module). The pinned substrate is agent-eval `>=0.97.0 <1.0.0`; the sandbox substrate that materializes profiles into harness shapes is `@tangle-network/sandbox` (peer `>=0.8.0 <1.0.0`). The neutral contract types (`AgentProfile`, `AgentProfileMcpServer`, `HarnessType`, `ReasoningEffort`, `Part`/`ToolPart`/`ToolState`) are owned by **`@tangle-network/agent-interface`** (peer `>=0.10.0 <1.0.0`) — the single source of truth. Substrate symbols (`selfImprove`/`gepaProposer`/`defaultProductionGate`/`heldOutGate`/`pairedBootstrap`/…) are re-exported through `@tangle-network/agent-eval/contract` (or `/campaign`), not local to this package.
>
> **`./loops` is the runtime barrel** — `package.json` maps it to `src/runtime/index.ts`. Everything below labelled `/loops` is the recursive-atom + loop-kernel surface.
>
Expand Down Expand Up @@ -55,7 +55,7 @@ Every symbol below is a LOCAL export of this package (subpath shown) unless tagg
| Run a sandbox coding rollout, round-synchronous (fresh box per round) | `runLoop(options)` — `/loops` | a `new Sandbox()`+acquire+stream+parse+delete loop, or a 2nd winner-selector |
| Run + **resume** ONE persistent box across turns | `openSandboxRun(client, opts, deliverable)` — `/loops` | a per-domain `new Sandbox`+`box.fs.read`+delete copy |
| Pick / register a leaf backend, or bring your own agent | `createExecutor({ backend })` / `createExecutorRegistry()` / implement `Executor` — `/loops` | a per-vendor adapter or closed `inline\|sandbox\|cli` switch (won't report through the `UsageEvent` channel) |
| Evolve a **prompt/string** surface | `gepaProposer({ llm, model, target })` (default inside `selfImprove`) — `agent-eval/contract` | a hand-rolled prompt-mutation reflection loop with its own Pareto bookkeeping |
| Evolve a **prompt/string** surface | `gepaProposer({ llm, model, target })` (default inside `selfImprove`; the skill-surface twin is `skillOptProposer`, same source) — `agent-eval/campaign` | a hand-rolled prompt-mutation reflection loop with its own Pareto bookkeeping |
| Self-improve a profile (one pluggable verb) — START HERE | `improve(profile, findings, { surface, gate })` — root `.` (the RSI verb; defaults the generator from `surface`, wraps `selfImprove`) | a bespoke optimize loop, or calling `selfImprove`/a skill-optimizer directly for the common case |
| Measure **one profile artifact's marginal lift** (with-vs-without, score+cost) / catalog artifacts | `measureMarginalLift(...)` / `ArtifactRegistry` (`applyArtifact` is the one `ArtifactKind`→`AgentProfile`-field bridge) — `/lifecycle` | a hand-rolled with/without ablation loop, or a per-kind `if kind==='skill'…` profile-field switch |
| Run the **whole artifact lifecycle** — generate→measure→promote→store→compose, then drift-watch/dedupe the live set — over ANY profile surface (skill/prompt/tool/MCP) | `runLifecycle({ baseline, generators, evalRunner, gate })` then `composeProfile(registry, base, query)`; maintain with `driftWatch(...)` / `dedupeArtifacts(...)` — `/lifecycle` | a per-surface improve loop, a hand-rolled promote→compose step, or re-running `measureMarginalLift` without the registry/gate spine. The ONLY per-surface code is a thin `CandidateGenerator` (`skillGenerator` distills, `promptGenerator`/`buildableGenerator` for the rest) |
Expand Down
6 changes: 3 additions & 3 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@tangle-network/agent-runtime",
"version": "0.74.0",
"version": "0.75.0",
"description": "Shared task-lifecycle skeleton for agents: a recursive loop kernel for chat turns, one-shot tasks, and multi-attempt loops, with trace capture and eval-gated self-improvement. Domain behavior lives in adapters; scoring and ship-gates in @tangle-network/agent-eval.",
"homepage": "https://github.com/tangle-network/agent-runtime#readme",
"repository": {
Expand Down Expand Up @@ -89,7 +89,7 @@
},
"devDependencies": {
"@biomejs/biome": "^2.4.15",
"@tangle-network/agent-eval": ">=0.95.0 <1.0.0",
"@tangle-network/agent-eval": ">=0.97.0 <1.0.0",
"@tangle-network/agent-interface": ">=0.10.0 <1.0.0",
"@tangle-network/sandbox": ">=0.8.0 <1.0.0",
"@types/node": "^25.9.3",
Expand Down Expand Up @@ -117,7 +117,7 @@
"license": "MIT",
"packageManager": "pnpm@10.28.0",
"peerDependencies": {
"@tangle-network/agent-eval": ">=0.95.0 <1.0.0",
"@tangle-network/agent-eval": ">=0.97.0 <1.0.0",
"@tangle-network/agent-interface": ">=0.10.0 <1.0.0",
"@tangle-network/agent-knowledge": ">=1.7.0 <2.0.0",
"@tangle-network/sandbox": ">=0.8.0 <1.0.0",
Expand Down
Loading
Loading