Track the Codex subscription credential lifecycle across ACP conversations

[simonrosenberg]

Context

Codex ChatGPT subscription credentials are mutable, rotating state. ACP conversations may run in separate Codex processes or sandboxes and may use independent writable credential copies.

OpenHands needs one credential lifecycle that propagates completed rotations and coordinates concurrent consumers across deployment topologies.

Work

• Phase 1: propagate rotated Codex credentials across sequential ACP conversations #4170
• Phase 2: coordinate concurrent Codex credential refresh across ACP conversations #4171

Deployment-specific issues and pull requests should link to the relevant phase.

[simonrosenberg]

Alternative Phase 1 implementation using a deployment-neutral generic credential broker: #4173. It targets the behavior in #4170 across standalone Agent Server, local Docker, and SaaS under the architecture direction in OpenHands/OpenHands#15298; it does not supersede the existing phase structure.

[simonrosenberg]

EVIDENCE

Production was investigated read-only using Datadog, GCloud Logging, and the prod-runtime Kubernetes context. Query window: 2026-06-01 through 2026-07-21 UTC.

Observed production incidents

• About 166 initial CODEX_AUTH_JSON materializations occurred across 164 distinct runtime pods after the production rollout began on 2026-06-30.
• GCloud Logging and Datadog both contain 17 occurrences of Failed to start ACP server: Invalid params.
• Those 17 events came from 15 distinct Codex runtimes/conversations. Two runtimes retried and failed twice.
• Every failed runtime:
  1. materialized CODEX_AUTH_JSON into its conversation-local CODEX_HOME/auth.json;
  2. initialized @agentclientprotocol/codex-acp 1.1.2;
  3. selected Authenticating with ACP method: chat-gpt; and
  4. failed 600.2-600.4 seconds later with Invalid params.
• From 2026-07-15 onward, 15 of 32 newly materialized Codex-auth runtimes later emitted this failure. Across the full rollout window, the raw distinct-runtime ratio was about 15/164. These are telemetry ratios, not unique-user impact rates: traffic was batched and several failures were repeat attempts.

UTC date	Initial Codex auth materializations	Invalid params events	Distinct failed runtimes
2026-07-15	10	5	4
2026-07-16	3	3	3
2026-07-17	1	1	1
2026-07-18	7	7	6
2026-07-19	7	1	1

The first matching failure was at 2026-07-15 06:51:40 UTC, before the originally reported incident.

Originally reported incident

Conversation 351ac142076b4d4d8a16dda25bf19bd1, runtime zbifrhphyeieeujj:

• 08:06:17.583 UTC - materialized CODEX_AUTH_JSON to the conversation-local Codex auth.json.
• 08:06:22.575 UTC - Codex ACP initialized and ChatGPT authentication began.
• 08:16:22.787 UTC - Failed to start ACP server: Invalid params.
• The runtime-local Codex application-server log captured for the original report showed a 401 with refresh_token_reused: the supplied refresh token had already been used to generate a successor.

The pod remained healthy and continued returning readiness responses throughout the authentication attempt. There was no ImagePull, scheduling, restart, OOM, routing, or pod-health failure.

The original pod has since been removed and its Deployment is scaled to zero. Fourteen of the other failed-runtime Deployments still existed during inspection, all scaled to zero and all configured with ghcr.io/openhands/agent-server:1.36.0-python.

Repeated attempts

Two conversations were retried immediately after the first ten-minute failure. In each case the SDK logged that the conversation-local CODEX_AUTH_JSON file was already present and was being preserved via seed-if-absent. Both retries initialized Codex, began ChatGPT authentication again, and failed after another ten minutes with the same Invalid params error.

On 2026-07-16, three runtimes started within seconds of one another and all three failed after ten minutes. This is evidence that all three received unusable credential state; it is not evidence of a concurrent refresh race with one successful refresh owner and stale losers.

The 2026-07-18 incidents likewise formed repeated chains of already-failing credentials rather than an observed one-winner/multiple-loser refresh sequence.

What the logs prove

The production evidence confirms:

• a repeated Codex ChatGPT authentication failure after conversation-local credential materialization;
• an exact and consistent ten-minute failure signature;
• repeat failures when the same conversation-local file is preserved; and
• one provider-confirmed refresh_token_reused incident.

The code path makes sequential staleness plausible: Cloud supplies the canonical CODEX_AUTH_JSON, the SDK materializes a writable per-conversation copy, Codex can rewrite that copy, and no current path writes the successor back to the canonical Cloud secret for the next conversation.

Evidence limitation

Neither Datadog nor GCloud Logging contains the provider-side refresh_token_reused reason; searches over the full window returned zero centralized matches. That reason was available only in Codex's runtime-local log for the original incident.

Therefore:

• the original incident's reused-token cause is confirmed;
• the other 14 runtimes are strongly consistent with the same failure mode, but their provider reason is not independently captured; and
• no production incident currently proves the separate Phase 2 concurrent-refresh race described in Phase 2: coordinate concurrent Codex credential refresh across ACP conversations #4171.

We should add sanitized auth-failure classification and credential-generation/writeback telemetry before attributing future Invalid params failures to token reuse or deciding that distributed refresh ownership is required.

No production resources were modified during this investigation.

[simonrosenberg]

Phase 1 implementation plan

This resolves the choice between the two Phase 1 implementation options — the deployment-neutral credential broker (#4173) and the minimal write-through (#4174) — after tracing the current SDK, Cloud, and agent-canvas code, the in-flight PRs (#4124, OpenHands/OpenHands#15287), and the production evidence above.

Decision

Land #4124 as the Phase 1 implementation, with the OAuth refresh proxy dark-launched, sanitized credential telemetry added in the same phase, and two #4170 criteria amended to match the reviewed behavior. Do not build the generic broker (#4173) in this phase. Do not re-derive a smaller implementation from #4124 (#4174 as written).

One ground-truth fact frames everything: the SDK materializes auth.json into a per-conversation directory in every deployment that supplies CODEX_AUTH_JSON, regardless of the isolation flag. The staleness topology is therefore intrinsic to SaaS and to canvas local Docker; only local-native use of the host's own codex login file escapes it. Phase 1 must cover both, and #4124 already does: the Cloud store behind OpenHands/OpenHands#15287 for SaaS, and the agent-server's own store behind the folded-in local broker (#4137) for canvas-local, where the ACP process and the canonical store are colocated.

Why not the generic broker (#4173)

• The deployment-varying behavior is two operations: resolve the current value with its version, and compare-and-swap replace. The difficult and dangerous code is elsewhere — the Codex-specific lifecycle (materialization, stable reads, masking timing, turn/close barriers, error typing) and deployment-specific authorization. Three review passes over the current implementation put every finding in those two layers and none in transport/identity, the only layer Phase 1 implementation option: deployment-neutral credential broker #4173 generalizes.
• There is one rotating consumer, with no second in sight: Claude Code uses a long-lived env token, Gemini's service-account key is static, and the supported provider set is fixed. Phase 1 implementation option: deployment-neutral credential broker #4173's real motivation is deployment families, not consumers; that is addressed below with a written contract rather than a hosted service.
• Design: reference-only credentials and safe runtime delivery OpenHands#15298 is not accepted. Minting /api/credential-broker/v1, UUID references, integer versions, a start_conversation signature change, persisted credential_bindings, and ensure_credential provisioning now would pre-commit shapes that design may still change — risking a legacy v0 broker plus a provisioning/migration layer with no Phase 1 user payoff. /api/internal/conversations/{id}/codex-auth is at least honestly labeled disposable.

#4173 remains the right shape for a later #15298 implementation slice, if that design is accepted. Two pieces of its discipline are pulled forward below so the eventual migration is a swap, not a rewrite.

Why not extract #4174 out of #4124

#4174's behavior already exists inside #4124 (approved, CI green, based on main), hardened by three review passes. The extraction is not a subtraction:

• The refresh-URL derivation is also the lifecycle factory gate; naive proxy removal silently disables all write-back.
• The ancestor-digest sidecar is the safety net for close()'s current flush-before-terminate ordering; removing it requires a shutdown reorder in exactly the code where review findings clustered.
• sync()'s control flow gets rewritten rather than trimmed, a double-digit number of tests die, and the result forfeits an approved branch for re-review of the riskiest layer — while still not matching Phase 1 implementation: minimal Codex auth write-through synchronization #4174's letter (no idle monitor; the digest is still the wire version; grant headers are still persisted on the resume rebind path).

The subtraction costs more than landing the reviewed implementation and produces neither spec. #4174 is repurposed below as the reconciliation record.

Landing plan

1. Merge fix(acp): persist brokered Codex subscription auth #4124 with the refresh proxy dark-launched. The production evidence above shows no incident attributable to the concurrent-refresh race: every multi-failure cluster was repeated consumption of already-invalidated state. Keeping the proxy code (zero deletion, zero re-review) while gating the CODEX_REFRESH_TOKEN_URL_OVERRIDE injection behind a default-off setting preserves Phase 1: propagate rotated Codex credentials across sequential ACP conversations #4170's non-goal at runtime, avoids coupling Codex token refresh to broker-route availability before that is needed, and leaves the remedy one flag away if telemetry ever proves the race. The injection is a single site in the lifecycle's bind(); the factory gating is unaffected. Write-through capture of rotations works identically with the proxy off.

2. Ship sanitized credential telemetry in the same phase. The investigation found the provider-side reason (refresh_token_reused) only in a runtime-local log; centralized logging had zero matches over seven weeks. Phase 1 adds: credential generation-transition events (digest-to-digest, never values), CAS outcomes (written / conflict / failed), and the typed auth-failure class, all centrally logged. This is the decision gate for Phase 2: coordinate concurrent Codex credential refresh across ACP conversations #4171 and turns the next incident from archaeology into a query. Separately, the 30-second authenticate timeout with typed ACPAuthRequired replaces the observed 600-second untyped Invalid params hang.

3. Amend two Phase 1: propagate rotated Codex credentials across sequential ACP conversations #4170 acceptance criteria explicitly rather than papering over them:

   • Background writes reach the store without another user turn → detection is event-driven at the post-authenticate, post-prompt, pre-masking, and close barriers; there is no idle monitor. The residual loss window is a hard crash during idle, already waived under Limitations.
   • The canonical version is the CAS generation; digests are not part of the remote interface → the wire version is an opaque store-issued token, which today is a content digest. Building integer versions now would churn approved code for cosmetics; the durable fix is item 5.

4. Resolve teardown authorization in fix(acp): broker concurrent Codex subscription auth OpenHands#15287 before merge. Flush-before-key-invalidation ordering is not implementable in the current pause flow: pause_sandbox commits the key invalidation before calling the runtime, there is no primitive to await an agent-server flush, and idle-reap pauses originate runtime-side. The options are (a) the hardened 30-second teardown grace key (DB-enforced expiry, codex-auth routes only, cleared on resume/delete) as the interim — permitted by Phase 1: propagate rotated Codex credentials across sequential ACP conversations #4170's teardown language — or (b) the structural fix: authorize the codex-auth routes by the scoped JWS grant alone, with conversation-scale expiry, independent of the sandbox session key, which removes the grace window and its migrations entirely. Direction (b) is the target; whether it lands inside #15287 or immediately after is a Cloud sequencing call. Independently of that choice, the grant lifetime drops from 14 days to conversation scale.

5. Pull two pieces of broker discipline forward as the durable seams:

   • Server-issued opaque version token. Today the client derives the CAS token by hashing credential content; that conflation is the one thing that would calcify. GET/HEAD/PUT respond with the synchronization version as an opaque token (the implementation may remain a digest); the client stops deriving it. This makes the eventual name-to-CredentialRef and digest-to-version migration a server-side swap. Fold into #15287's revision or land as the first follow-up; required before any third deployment family is built.
   • A written authorization contract for the third family. OSS app-server per-sandbox deployments remain unfixed: those sandboxes have a reachable canonical GET but no CAS PUT, and the agent-server-local broker CASes a non-canonical copy there. Reopen Support concurrency-safe local Codex auth writeback #4161 (or file a successor) scoped to that topology, specified as a port of the same wire contract — GET/HEAD/PUT, opaque version, no-store, fail-closed CAS — with grant-style authorization, so it is a port rather than a third bespoke security design.

6. Re-scope Phase 2: coordinate concurrent Codex credential refresh across ACP conversations #4171 as telemetry-gated. No distributed refresh-ownership work until the item 2 telemetry attributes a real one-winner/stale-losers refresh race in production. Concurrent sessions remain allowed; the dark-launched proxy is the prepared remedy.

Issue bookkeeping

• Phase 1: propagate rotated Codex credentials across sequential ACP conversations #4170: criteria amendments from item 3.
• Phase 2: coordinate concurrent Codex credential refresh across ACP conversations #4171: telemetry-gated per item 6.
• Phase 1 implementation option: deployment-neutral credential broker #4173: closed for Phase 1; revisit as an Design: reference-only credentials and safe runtime delivery OpenHands#15298 implementation slice if that design is accepted.
• Phase 1 implementation: minimal Codex auth write-through synchronization #4174: repurposed as the reconciliation record — the deltas the merged implementation still owes (shutdown stop-then-flush reorder, runtime-only activation on resume, server-issued version token) tracked as follow-ups.
• Support concurrency-safe local Codex auth writeback #4161: reopen or supersede, scoped to the OSS app-server per-sandbox family under the item 5 contract.

[simonrosenberg]

Proposed Phase 1 implementation plan

Decision

Implement #4174's narrow VersionedCredentialBinding and the complete synchronization lifecycle specified by #4170.

Do not make Phase 1 depend on the generic credential broker in #4173, the complete reference-only secrets architecture in OpenHands#15298, or the distributed refresh-owner coordination in #4171.

This is the middle path between two unsuitable extremes:

• A post-turn/post-close file copy is too weak. Codex can rewrite auth.json during startup, authentication, a prompt, background work, or shutdown. It would also miss partial writes, masking order, CAS conflicts, and process-exit ordering.
• The full credential architecture is a reasonable destination, but it adds stable credential identities, catalog APIs, storage migrations, grants, revocation, audit, several hosting topologies, and broader persistence/UI work. Those primitives are not required to repair the sequential rotation path, and they still would not prevent two Codex processes from presenting the same refresh token to the provider.

The narrow binding preserves the important architectural boundary: the ACP integration depends on a versioned credential authority, not on OpenHands storage details. A future generic broker can implement the same protocol without changing the Codex synchronizer.

The production evidence does not change this direction. It makes two constraints especially important:

1. Only the originally inspected runtime has a provider-confirmed refresh_token_reused result. The other failures are consistent with the same mechanism but do not prove it. Phase 1 needs sanitized generation/writeback/authentication telemetry.
2. The simultaneous failures do not demonstrate a one-winner/multiple-loser refresh race. Phase 2 coordination should remain separate until telemetry demonstrates that race.

Phase 1 outcome

The primary acceptance scenario is:

1. Conversation A resolves canonical CODEX_AUTH_JSON = R0, generation N.
2. Codex changes its private auth.json to R1.
3. The runtime validates and conditionally replaces (N, R0) with (N+1, R1) in the canonical store.
4. Isolated conversation B resolves R1 and starts from it.

Phase 1 must support both OpenHands Cloud and direct/local Agent Server use, including Agent Canvas. If Cloud is shipped first because of production urgency, call it Phase 1a and retain a Phase 1b issue for the local adapter rather than treating the cross-deployment issue as complete.

1. Minimal versioned binding

Add a provider-neutral runtime protocol along these lines:

@dataclass(frozen=True)
class ResolvedCredential:
    value: str
    version: str

class VersionedCredentialBinding(Protocol):
    async def load(self) -> ResolvedCredential: ...
    async def replace(self, expected_version: str, value: str) -> str: ...

The binding is already authorized and scoped to exactly one credential. It deliberately has no credential-name argument, catalog, discovery API, general CRUD API, or persisted reference model.

Contract requirements:

• version is an opaque generation owned by the canonical store.
• replace is an atomic compare-and-swap and returns the successor generation.
• A missing, deleted, revoked, unauthorized, or conflicting credential produces a typed error.
• A conflict is never resolved by adopting either side or by overwriting the canonical value.
• Retries are allowed only while the result is unambiguous. After an ambiguous transport failure, the adapter may reload and treat an exact value match as convergence; any different value fails closed.

Use a fresh random opaque generation stored alongside the encrypted value, or a monotonically increasing revision that cannot reset after delete/recreate. Do not use a plaintext SHA digest as the canonical version. A restored value can repeat, and deletion/recreation would create an ABA problem. A SHA digest remains useful only inside the runtime as a local file-change detector.

2. Runtime-only activation

Introduce an internal RuntimeActivation that supplies the exact binding to the conversation/ACP runtime. It must be an in-memory, non-Pydantic object and must not appear in:

• StartConversationRequest.secrets
• SecretRegistry
• agent settings
• events
• conversation metadata or state
• snapshots
• serialized retry/resume data

The activation is attached through the trusted launch path after durable request deserialization. A cold resume receives a new activation before Codex starts. Endpoint details and bearer grants are never reconstructed from conversation state.

For Cloud, OpenHands should issue a short-lived grant restricted to the exact user, organization, credential, conversation, runtime, and load/replace actions. The Agent Server turns that grant into an HTTP binding in memory. Responses must use Cache-Control: no-store, and request/response bodies must be excluded from application logs, traces, and error reporting.

For standalone Agent Server, construct an in-process binding over the existing file secret store. There is no reason to introduce a local HTTP broker when the canonical store and ACP process share the Agent Server trust boundary.

CODEX_AUTH_JSON remains the existing canonical credential identity in Phase 1. Do not migrate users to a new CredentialRef or add a new credential table as a prerequisite.

3. Runtime-private materialization

Every Codex process launch, including a cold resume, must begin from a fresh canonical resolution:

1. Attach a fresh runtime activation.
2. Load and validate the canonical ChatGPT auth document.
3. Register its document and token fields with the runtime-only masking layer.
4. Create a fresh 0700 runtime directory outside conversation persistence/snapshots.
5. Atomically materialize CODEX_HOME/auth.json with mode 0600.
6. Start the file monitor before Codex can write.
7. Spawn Codex.

Do not preserve an old conversation-local auth.json with seed-if-absent. The production retries show that this preserves unusable state and repeats the ten-minute failure. On a controlled in-process restart, complete the final synchronization barrier before discarding the old runtime directory, then resolve canonical state again.

Recovery of an unsynchronized file left by SIGKILL, node loss, or another hard crash remains an explicit Phase 1 non-goal.

4. Continuous synchronizer

The monitor is the primary synchronization mechanism; lifecycle barriers make it deterministic at externally visible boundaries.

For each possible file change:

1. Read a stable snapshot, for example unchanged metadata around the read or two matching reads.
2. Tolerate transient empty, truncated, and partially replaced files with bounded retry.
3. Accept only a complete valid ChatGPT auth document.
4. Compare a runtime-local digest with the last accepted document; unchanged content is a no-op.
5. Add the complete document and all token-bearing fields to an in-memory masking set before any later ACP output can be persisted or surfaced.
6. Enqueue one serialized replace(expected_version, value) operation.
7. On success, advance the expected canonical generation and accepted local digest.
8. On conflict, deletion, revocation, authorization failure, or exhausted writeback failure, enter a sticky failed state.

The dynamic masking values must be transient. Do not put rotated plaintext into a serializable SecretRegistry. If the existing masking implementation cannot accept non-persisted values, add a runtime-only masking overlay and apply both the durable registry and transient overlay before event persistence.

Synchronization barriers are required:

• after ACP authentication;
• after each prompt/turn boundary;
• before every ACP output/event is persisted or surfaced;
• before returning a successful pause/stop;
• during shutdown.

A barrier samples the file, completes masking registration, drains any queued canonical replacement, and raises the sticky synchronization error before the caller can continue.

Clean shutdown ordering is:

1. Quiesce/stop Codex.
2. Wait for the process to exit.
3. Perform a final stable read with bounded retry.
4. Complete the final CAS and drain the writer.
5. Stop the monitor.
6. Discard the activation and runtime directory.

5. Canonical store adapters

OpenHands Cloud

Extend the existing encrypted custom-secret authority with a canonical generation and exact-row operations:

• load CODEX_AUTH_JSON plus generation;
• conditionally replace it when the generation matches;
• atomically assign a new generation on every user- or runtime-originated update;
• return conflict/not-found/revoked distinctly.

All existing settings/UI writes that can update or delete CODEX_AUTH_JSON must go through the same generation-changing path. A bulk replace path must not delete/reinsert or silently overwrite this row while a runtime holds a generation. This can be implemented by reserving this key from the legacy bulk operation and routing it through versioned per-key operations; it does not require a generic credential service.

OpenHands currently resolves custom secrets into the ACP start request. For this key, stop inserting plaintext into durable request.secrets; attach the runtime activation out of band instead. Other secret behavior can remain unchanged in Phase 1.

Standalone Agent Server / local Agent Canvas

Extend the file-secret representation with an opaque generation. Under the existing file lock:

• assign a generation to a legacy entry once;
• atomically replace value and generation in the same file update;
• give deletion/recreation a new generation;
• implement the binding directly over these operations.

Agent Canvas already stores CODEX_AUTH_JSON through its normal secret UI. In direct Agent Server mode it sends a LookupSecret back to that same Agent Server. The server can recognize the runtime-managed Codex credential, move it into an in-process activation, and omit the plaintext/binding data from durable conversation state. No new Canvas credential-management UI or generic reference model should be required.

6. Failure model

Expose actionable failures rather than the current generic Invalid params result:

• needs_reauthentication: canonical credential is absent or initially invalid;
• credential_update_conflict: another writer changed or deleted the canonical credential;
• credential_activation_expired or credential_access_revoked: the runtime is no longer authorized;
• credential_sync_failed: a local validation, monitoring, or writeback failure makes state uncertain;
• classified ACP authentication timeout/provider rejection where available.

After a sticky synchronization failure, do not emit further normal ACP output or continue using the credential. Never surface raw provider responses if they can contain credential material.

7. Sanitized observability

Ship observability with Phase 1, not as a later cleanup. Record counters/spans for:

• canonical load success/failure;
• initial materialization;
• valid local rotation detected;
• conditional replace attempted/succeeded/conflicted/failed;
• barrier and final-flush outcomes;
• prohibited stale local-file reuse;
• authentication stage, elapsed time, and sanitized failure classification.

Safe dimensions include deployment mode, provider, component version, operation, and result. If cross-runtime generation correlation is necessary, use a non-secret keyed identifier. Never log or trace:

• the auth document or individual tokens;
• a plaintext file digest;
• activation endpoints or grants;
• authorization headers;
• raw provider bodies.

Use this telemetry to validate that canonical writeback removes the sequential failure pattern. Do not infer a Phase 2 concurrent-refresh incident merely from simultaneous failures; require evidence of one successful refresh owner and stale concurrent losers.

8. Implementation and rollout sequence

1. Store preparation: Add canonical generations and exact load/CAS operations to Cloud and local stores. Keep existing read paths backward compatible.
2. SDK/Agent Server support: Add the binding protocol, runtime activation plumbing, transient masker, synchronizer, barriers, typed errors, in-memory test adapter, and optional activation support. Without activation, existing non-Codex behavior remains unchanged.
3. Cloud activation: Issue scoped runtime grants, construct the HTTP binding, and omit CODEX_AUTH_JSON from durable ACP secrets only when the target Agent Server advertises support.
4. Local activation: Construct the binding in process and cover direct Agent Canvas create/resume behavior.
5. Canary: Enable for a small Cloud cohort, verify no material/grant persistence and monitor generation/writeback/authentication telemetry.
6. General rollout: Expand Cloud, then local deployments. Remove the legacy Codex seed-if-absent path after mixed-version rollout is complete.

The mixed-version order matters: deploy store/API support first, then Agent Server capability, then enable OpenHands activation/secret stripping. An old Agent Server must never receive a request where the only usable credential was removed.

I would extract narrowly useful atomic-materialization, validation, masking, and test code from #4124 and OpenHands#15287, but not merge either broad implementation wholesale. Digest CAS, refresh proxying/Redis ownership, remote polling/adoption, ancestor recovery, persisted grant-bearing LookupSecret, and teardown migration are outside or conflict with this Phase 1 contract.

9. Required tests

SDK unit/contract tests

• initial canonical load and atomic private materialization;
• stable-reader behavior across truncate/write and atomic rename patterns;
• invalid/partial JSON never published;
• unchanged content produces no replace;
• serialized writes advance the returned canonical generation;
• conflict, deletion, revocation, and exhausted failure become sticky;
• transient masks are installed before output and excluded from serialization;
• ambiguous replace response converges only when canonical bytes match;
• stop process before the final read/CAS/drain;
• fresh launch/resume never preserves the previous runtime file.

Persistence/security tests

Scan request models, stored metadata, state, events, snapshots, logs, and traces to prove that none contains:

• CODEX_AUTH_JSON plaintext;
• old or rotated tokens;
• the runtime grant;
• binding endpoint/authentication data;
• transient masking values.

End-to-end tests

• Cloud A starts with R0, rotates to R1, and isolated B starts with R1.
• Direct/local Agent Canvas performs the same A-to-B sequence.
• Rotation during authentication followed by immediate pause is retained.
• A background write without a subsequent prompt is retained.
• A conflict fails closed and does not overwrite the winner.
• Cold resume obtains a fresh activation and canonical generation.

A two-runtime test should prove only canonical-store safety: if both load generation N, at most one different replacement can commit. It must not claim to provide upstream refresh ownership. Preventing both runtimes from using the same provider refresh token is Phase 2.

Explicit Phase 1 non-goals

• hard-crash recovery after an unflushed local rotation;
• distributed/provider-side refresh ownership;
• OAuth refresh proxying or CODEX_REFRESH_TOKEN_URL_OVERRIDE;
• Redis locks;
• remote polling, remote state adoption, or three-way merging;
• recovery from ancestor conversation files;
• durable retry across runtime loss;
• a generic credential catalog or new stable CredentialRef model;
• broad Tavily/MCP/org-secret migration;
• general release/revoke APIs created solely for runtime teardown.

This plan fixes the observed architectural gap without committing Phase 1 to the complete secrets redesign. It also leaves a clean evolution path: the future credential broker becomes another VersionedCredentialBinding implementation, while the Codex materialization, monitoring, masking, CAS, barriers, and shutdown behavior remain unchanged.