Release 0.18.8 — errno-108 array-fanout robustness#65
Merged
Conversation
f.readlines() returns lines that already end in \n, so '\n'.join(...) inserts a second \n between every line. YAML treats \n\n as a literal newline (vs \n folding to a space), so any plain scalar that yaml.dump wrapped past 80 cols parsed with literal \n in the value via this path but correctly via plain yaml.safe_load. Net effect downstream: type-description strings diverged between data_types/*.yml (regular load) and transforms/*/_metadata/types/*.yml mirrors (this load path), producing different type hashes and breaking DataInstance->Transform binding for any long-described type. Reported from spanish-lakes/metagenomics: ref::kraken2_db and ref::metaphlan_db hashed differently across paths. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
5d2aa2f's fd-5 ctl channel wedged every Agent.Deploy() against an ssh home: fd 5 doesn't traverse ssh, so the trampoline's `printf >&5` on the remote bash wrote to a closed fd and no completion frame ever arrived. Replaces the fd-5 + JSON-frame model with per-Exec inline markers on stdout AND stderr: <user_cmd> __rc=$?; printf '\x1eMSM_END_<TOKEN>_<NONCE> %s\x1e\n' "$__rc"; \ printf '\x1eMSM_ERR_<TOKEN>_<NONCE>\x1e\n' >&2 The marker is just bytes. Whichever shell is parsing bash's stdin when it reaches the line emits the marker, and ssh / nested bash / docker exec / `bash -c` forward the bytes back unchanged. This is the single mechanism that generalises across local + sub-shell cases without pre-installed state inside the target shell: - Exec("ssh host") — marker comes back from remote - Exec(cmd) after entry — runs on remote (state persists) - Exec(cmd) local — marker on local bash - Exec(rsync src ssh://...) — rsync uses its own pipes, marker runs on local bash after rsync exits Collision immunity: \x1e (ASCII RS) brackets the marker, plus a per-LiveShell 128-bit token and a per-Exec random nonce. The strip- from-delivery rule only fires when `nonce in self._pending`, so user output containing the marker shape with an unknown nonce passes through verbatim — keeping the G2 collision-immunity tests green. Drops the ctl pipe (`_ctl_r`, `_ctl_reader`, `_ctl_fd_in_child`, `_on_ctl_line`, `extra_pass_fds`), the `import json`, the `__msm_done` bash function injection, `_install_trampoline` / `RemoteExec`, and the `__msm_rc=$?; (exit $__msm_rc)` gymnastics in ExecAsync. TerminalProcess keeps `extra_pass_fds` as a default `()` param for back-compat; nothing uses it now. models/remote.py: - `_execute_ssh` probe phase: dropped the `; echo exited` workaround (it was always broken — `echo exited` runs whether ssh succeeds or fails); use `Exec(f"ssh {remote}")` + `res.exit_code` instead. - `_join` phase: interactive ssh entry per batch (`Exec(f"ssh {dest_host}")`), file checks on the now-remote shell, then `Dispose()` — no `Exec("exit")`. This sidesteps two failure modes discovered while testing against ssh xpsz: 1. `Exec("ssh host 'cmd'")` (non-interactive command-mode ssh) wedges: ssh's non-interactive remote command doesn't read stdin, so the trailing marker bytes are forwarded to the remote one-shot which discards them. 2. `Exec("exit")` from a sub-shell wedges: the sub-shell terminates before the marker emission line is read. End-to-end verified: `Agent.Deploy(ssh://xpsz/...)` runs to completion through interactive ssh + multiple Exec / rsync / per-batch existence check / container-side `deploy_from_container` + relay extraction. `pytest tests/test_live_shell.py -v` — 18/18 pass (G1–G8 + strict- ordering invariant + batched-async). Known residual limitations (in-band approach inherent): - `Exec("exec something")` replaces bash; no in-band approach survives. Refuse at API level if needed. - `Exec("cat")` / `Exec("read x")` — user command consumes the marker line from stdin. Caller uses `timeout=`. - `Exec("exec 2>/tmp/log")` — stderr marker goes to the file forever. Not handled. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The Approach D rewrite let `Exec("ssh host")` work cleanly because the
marker bytes flow through ssh to the remote shell and back. The mirror
case — leaving the sub-shell via `Exec("exit")` — wedged: bash writes
`exit\n` plus marker emission as one stream, ssh greedily consumes both,
the remote bash exits before reading the marker, and the marker bytes
are lost in transit.
SubShell(entry_cmd) is the surface API for shell-boundary crossing:
- __enter__ runs Exec(entry_cmd); the marker comes back from the
sub-shell (no special path needed).
- __exit__ runs _pop(): writes `exit` alone, waits for true output
quiescence via a multi-sample idle detector (N consecutive samples
where now - _last_byte_time >= quiescence_ms, with a floor), THEN
writes a fresh marker emission that lands on the now-foreground
parent shell. If the first marker is lost (sub-shell still draining
when written), one retry recovers — lost markers are no-op on a dead
remote.
LiveShell gains _last_byte_time (updated in _make_tee on every non-empty
chunk) and _depth (bookkeeping for nesting). A class attribute
_pop_drop_first_marker is a test seam that forces the retry path; off
by default.
models/remote.py:_join now uses one shared LiveShell across all
dest-host batches, entering via `with check_shell.SubShell(f"ssh {host}")`
per batch instead of creating + disposing a fresh LiveShell per batch.
Local-dest branch goes back to plain Path.exists().
Verified:
- pytest tests/test_live_shell.py: 25/25 green (18 prior + 7 new
covering nested-bash round-trip, two-level nest, exception-in-body,
time bounds, chatty quiescence, retry path, timestamp update).
- e2e deploy sockeye (LiveShell + APPTAINER + scratch home): exit 0;
lib/agent.yml and relay/msm_relay landed at
/scratch/st-shallam-1/txyliu/metasmith_ws/msm_subshell_test/.
- e2e deploy fir (LiveShell + APPTAINER + home): exit 0; same artifacts
at /home/phyberos/metasmith_ws/msm_subshell_test/.
Version bumped to 0.18.4 to signal completion.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
….4 (Bug E.4 fir) The SIF→sandbox auto-unpack added for Bug E.2 (commit 96c4634) gated on a single proxy: `[ -u .../starter-suid ]`. That conflated two distinct conditions — "host needs squashfuse workaround for SIF" and "sandbox path is itself safe on this host" — and shipped a sandbox dir on hosts where the sandbox path is the unsafe one. Empirically (today, fir login3 + cvmfs apptainer 1.3.5): array tasks routed through the sandbox SIGBUS 1/8 in the minimal repro; matches the ~30 FAILED of the original nextflow run on runs/qZdfQy2h. The reporter's parallel workaround (`mv sandbox sandbox.disabled_for_test_*`) has the SIF fallback succeeding on the same compute fabric. Mechanism, verified across four hosts by inspecting /proc/mounts and pgrep squashfuse while `apptainer exec` is live: - Sockeye 1.3.1 + setuid → SIF kernel-mount, sandbox kernel-overlay. Both safe. Verdict: use-sif (no sandbox built). - fir 1.3.5 no setuid → SIF squashfuse_ll (works under sbatch), sandbox fuse-overlayfs (SIGBUS under array contention). Verdict: use-sif (no sandbox built). - Cosmos 1.4.2 no setuid, WSL2 6.6 → SIF squashfuse_ll (Bug E.2 wedge under relay fork chain), sandbox kernel-overlay. Verdict: use-sandbox. - xps 1.4.5 no setuid, WSL2 6.6 (the original Bug E.2 host) → same as Cosmos. Verdict: use-sandbox. The decision matrix is two-axis: setuid starter-suid AND apptainer major.minor. Static signals; no apptainer exec required at probe time. - Container.MakeSandboxDecisionProbe replaces MakeNeedsSandboxProbe. Emits `use-sif` or `use-sandbox`. setuid present → use-sif. Else major>=2 || (major==1 && minor>=4) → use-sandbox; else use-sif. Empty for non-APPTAINER runtime. - Agent.Deploy(): probe → capture VERDICT → build sandbox idempotently on use-sandbox branch (`[ -d ... ] || apptainer build`); else `rm -rf` any stale sandbox dir so a flipped verdict self-heals on next deploy. The `assertive=True` `rm -rf` prefix is preserved. - Container.MakeRunCommand(local=True) ternary is unchanged. Directory presence on the target host remains the run-time signal; deploy controls the presence. - libraries.py:1383 cache check `( [ -e sif ] || [ -d sandbox ] )` unchanged; on use-sif hosts the OR collapses to the SIF check. - AGENTS.md updated: new section heading "Apptainer SIF ↔ sandbox decision" describing the two-axis probe and what each verdict does. Tests rewritten: - tests/models/test_container_sandbox.py TestProbe → TestSandboxDecisionProbe; pins setuid axis, version axis, the two verdict literals, and the empty-for-docker contract. Existing TestCachePaths / TestBuildCommand / TestRunCommandSwitch unchanged. - tests/test_deploy_sandbox_step.py rewritten to pin the new VERDICT capture, the branch on `use-sandbox`, idempotent `[ -d ...]` guard, and the else-arm `rm -rf` for the stale-sandbox case. `assertive` test preserved. 15/15 sandbox+deploy tests pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
… markers
ExecAsync wraps the user cmd in `{ ...; } </dev/null` by default so children
(one-shot ssh, cat, read, sudo without -n, ...) inherit /dev/null as stdin
and cannot greedily consume the marker emission line off bash's stdin pipe.
This closes "known limitation hallamlab#1" from 7a8008f: Exec("ssh host cmd") and the
compound Exec("ssh host mkdir && rsync src host:dst") used by
main/local_mock/rsync.py wedged because ssh shipped the marker bytes to the
remote one-shot, where they were discarded; AwaitDone then waited on a
marker that would never arrive.
Diagnosed via /tmp/ls_probe.py against sockeye: Exec("ssh sockeye echo
from-remote") returned the expected stdout but rc=None at 30s. Confirmed
in-process: bash subshell parked in pipe_read at fresh prompt, no rsync/ssh
children, Python main thread futex-blocked in AwaitDone, NBR threads idle.
Brace group is the right wrap (not subshell parens) — it preserves env
mutations across Execs (e.g. `module load apptainer/1.3.1` in setup_commands
persists into later Execs on the same shell). `__rc=$?` on the next line
still captures the user cmd's exit code unchanged. body.rstrip() + explicit
'\n' before '}' guarantees the closing brace is its own token even for
multi-line bodies (agents.py:517's f"""...""" pattern).
inherit_stdin=True opt-in for cmds that DO need bash's real stdin — the
sub-shell-entry mechanism where the marker line is supposed to travel
through to the inner shell:
- SubShell.__enter__ (terminals.py)
- agents.py:151 — `shell.Exec(f"ssh {host}")` (Deploy's interactive ssh)
- remote.py:432 — `remote_shell.Exec(f"ssh {remote}")` (rsync _join batch)
Tests:
- 4 new local-only (mutation gates): stdin-stealing cmd doesn't block
subsequent Execs; explicit `< file` overrides the wrap; multi-line
body preserves env mutation across Execs; inherit_stdin=True passes
bash's stdin through to a child bash.
- 3 new network-gated (LIVESHELL_REMOTE_HOST=sockeye): ssh one-shot
rc=0, rc=1, and the exact ssh+rsync compound from rsync.py.
- conftest registers the `network` marker so pytest doesn't warn.
- test_pop_quiescence_under_chatty_output updated to pass
inherit_stdin=True on its bare `Exec("bash")` entry (mirroring what
SubShell.__enter__ now does internally).
Verified:
- pytest tests/test_live_shell.py: 29 local pass, 3 network pass against
sockeye.
- /tmp/ls_probe.py test B (Exec("ssh sockeye echo from-remote")): rc=0
in 0.3s (was: rc=None at 30s timeout).
- dev.sh -td sockeye: exits 0, sockeye-side terminals.py md5 matches local.
- Agent.Deploy(home=ssh://sockeye/...) end-to-end: msm, lib/agent.yml,
lib/msm_bootstrap, relay/msm_relay all land on sockeye with expected
sizes; relay binary md5 matches local target/.../msm_relay.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The previous comment ("remove work dirs of completed tasks to save
disk/inodes") described the opposite of what the setting actually does.
Reword to state why we keep the dirs (recoverable .command.err/.out for
failed tasks).
…g + yaml fix Patch release rolling up five dev commits: - yaml_safe_load no longer doubles newlines on round-trip (corrupts wrapped scalars) - LiveShell: inline-marker completion protocol survives sub-shells - LiveShell: SubShell(entry_cmd) context manager + Approach D rewrite, bumping version.txt to 0.18.4 - Apptainer SIF↔sandbox decision now probes at deploy: prefer SIF, build sandbox iff apptainer>=1.4 without setuid (Bug E.4 fir SIGBUS) - LiveShell: stdin-isolate user commands so ssh/cat/read can't steal markers - slurm.nf: clarify cleanup=false comment No new features — bug fixes and robustness only. release-alpha (0.19.0) remains separate. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
Author
Live smoke deploy against
|
Contributor
Author
Live smoke deploy against
|
| Host | apptainer | setuid | Probe verdict | Sandbox built? | Bug avoided |
|---|---|---|---|---|---|
| xpsz | 1.4.5 | no | use-sandbox |
yes | E.2 (WSL2 squashfuse_ll wedge) |
| fir | 1.3.5 | no | use-sif |
no | E.4 (fuse-overlayfs SIGBUS race) |
Both branches of the new probe exercised end-to-end on real hosts.
No build script consumes it: dev.sh -bs builds the SIF from the local Docker image via docker-daemon://, and no other code references the file. The def was also stale — it still pinned metasmith-0.17.1 and a dangling globusconnectpersonal copy path. Same conclusion reached independently on feat/audit (4fe238a, S5 of #344).
0.18.4 shipped with 3 of 4 relay binaries as 28-byte
"#!/bin/sh\necho 'stub relay'" placeholders because
--update_container chains -bp/-bd/-ud/-bs without -br and the cross-
compile target/ carried stale stubs. The Dockerfile COPY happily
ships them, and -ud/-bs had no content check, so the broken image
landed on quay.
_assert_real_relays spins the just-built image once, checks each
/app/msm_relay.{arch}-{sys} for the right magic (ELF for -linux,
Mach-O for -darwin) and a >100 KB size floor. Gated on -ud and -bs.
-bd stays unblocked so local debugging of partial builds is fine.
MSM_SKIP_RELAY_CHECK=1 overrides for emergency patches.
Also fixes -brc/-br: they referenced ./pack.sh which doesn't exist;
the real scripts are main/relay_agent/dev.sh -bb (cross-compile
container) and -b (cargo build all 4 targets). The error message
points at the now-working commands.
Verified: stub image (0.18.4-6e7020f) → rc=1 with the 3 bad slots
named; all-stub (0.18.4) → rc=1 with all 4; clean image
(0.18.3-dde0eec) → rc=0; MSM_SKIP_RELAY_CHECK=1 → rc=0.
examples/ is a self-contained agnostic test library — the smallest valid
metasmith library, reusable for any deploy/runtime smoke. Ships:
data_types/{examples,containers}.yml, metasmith.oci, echo_greeting.py,
and the compiled _metadata/ from `metasmith build -t data_types -r .`.
main/local_mock/smoke_hpc_deploy.py mirrors docs/source/setup/deployment.rst
verbatim — Agent(home=SshSource(...)).Deploy()+Generate+Stage+Run+Wait
against examples/, with the only host-aware bit being `module load apptainer`
in setup_commands (which IS in the docs). Anything more a host needs is a
documentation gap to file, not to paper over here.
Used by the feat-hpc-deploy-smoke scope to verify Bug E.2/E.4 fixes on
sockeye/fir/mira from a vanilla user's perspective.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The bump-procedure code block didn't mention -brc/-br at all, which is how 0.18.4 shipped with stub relays. List them explicitly and call out that --update_container skips -br. Document the _assert_real_relays gate on -ud/-bs and the MSM_SKIP_RELAY_CHECK=1 escape hatch.
Brief section explaining the layout, how to use it from a smoke script, and how to rebuild after edits. Lands alongside the examples/ directory introduced in 379c6ef. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…mples/ smoke library
phy0x1a79ed
changed the title
Contributor
Author
|
Updated to 0.18.5 (corrective re-release of 0.18.4). 0.18.4's published quay image shipped 28-byte stub relay binaries for 3 of 4 arches because the publish path used
|
…leged userns) The conda-forge apptainer build has no setuid starter-suid, so its sandbox path needs unprivileged user namespaces. Hosts that restrict them (Ubuntu 23.10+ apparmor_restrict_unprivileged_userns=1) fail Deploy() with "Operation not permitted". Verified on mira (RED, blocked) vs sockeye (GREEN, setuid module → use-sif). Document the failure signature and the three remedies: system setuid apptainer, enable userns, or use Docker. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Delete dead/unreferenced files surfaced during a cleanup audit: - plans/harmonize-data-instance-arity.md (stale design doc, 0 refs) - lib/data_types/hallamlab.yml (unreferenced type YAML) - main/transforms/test.py (broken: loads non-existent lib/transforms/...) - src/metasmith/testing/test_batched_bounce_fir.sh (orphan SLURM/apptainer debug script) Drop the now-pointless !/lib/data_types exception from .gitignore (keep !/lib/local). examples/, src/metasmith/testing/ (.py modules), and lib/local/ left intact - all load-bearing. pytest --collect-only: 435 tests collect clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_wait_for_init waited on a single fixed _INIT_TIMEOUT=5s total deadline and never checked whether bash was still alive, so a slow-but-fine bash and a dead bash were indistinguishable — both hit the ceiling and raised. Convert the wait to an inactivity loop keyed on _last_byte_time (the signal _make_tee already stamps on every chunk, the same one _pop samples for quiescence): succeed on both markers, fail fast the moment bash is observed exited (new TerminalProcess.IsAlive/ExitCode), and otherwise fail only after a full idle window of silence. Repurpose _INIT_TIMEOUT (now 300s) as the max idle window, not a total ceiling; add _INIT_POLL_INTERVAL for re-sampling. The healthy path still wakes on the marker's notify_all with no added latency. Adds G9 tests: activity survives past the idle window; dead bash fails fast with the exit code named. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add Container._store_root() as the single point of control for the
apptainer image-store location: ${APPTAINER_CACHEDIR:-<agent_home>/container_images}.
GetLocalPath/GetSandboxPath derive from it, so the pull (write), sandbox
build, and exec (read) sides all resolve the same root and the .sif/.sandbox
stay siblings. The value is a shell expression expanded on the execution
host (like $AGENT_HOME), so HPC deploys pick up the cluster setting; when
the var is unset, behavior is unchanged.
Reconcile the e2e spoof harness (it sets APPTAINER_CACHEDIR under the
sandbox home) so it pre-places the sif where deploy now looks, exercising
the override end-to-end. Update unit tests and docs (deployment.rst note,
AGENTS.md cache-layout).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… apptainer image store
Single-point Container._store_root() resolves the image store to
${APPTAINER_CACHEDIR:-<agent_home>/container_images}; falls back to the
agent-home default when unset. Pull/build/exec all derive from it.
Carries the 0.18.5 release commits (version bump + dev->release merge +
install-docs warning) since the branch was based on release.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…dow) Refine the init wait so liveness — not a clock — is the operative gate, per the load analysis: the deploy false positive came purely from a too-tight time deadline racing the reader threads for the GIL, and poll() (waitpid WNOHANG) can never report a running process as exited, so liveness is a safe gate that can't manufacture a false "dead" verdict. _wait_for_init now: succeeds on marker sync (checked first); fails fast the instant bash actually exits, naming the exit code (re-checking sync once to defend the emit-then-exit race); and falls back to a far ABSOLUTE backstop (_INIT_TIMEOUT=300s, elapsed from start) only for a wedged-but-alive shell. Replaces the previous inactivity window keyed on _last_byte_time, which was moot under load (that timestamp is stamped by the very reader thread that starves) and only obscured intent. The command/transform path is untouched: AwaitDone(timeout=None) still waits forever through arbitrarily long silence (transform protocols, nextflow, staging). Pinned by a new test. Tests (G9): dead->fast-fail; alive-but-slow->not killed (deploy-load case); wedged-but-alive->backstop fires; command-path timeout=None survives silence. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…test Replace the mocked-marker G9 init tests with real-process equivalents that rewrite the spawned argv (sleep N; exec bash / exit C / exec sleep N) so a real bash is genuinely slow, dies, or wedges — reproducing the deploy condition directly instead of poking internal bookkeeping: - slow_but_alive_bash_still_inits: marker delayed 6s (> old 5s) -> still inits. - dead_bash_fails_fast_with_rc: bash exits 7 before responding -> fast-fail naming rc. - wedged_alive_bash_hits_backstop: alive-but-silent shell -> far backstop fires. - command_path_timeout_none_survives_silence: unchanged (kept). - init_survives_reader_starvation_under_load (@slow): GIL contention starves the reader threads while constructing many real LiveShells; all must come up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fold METASMITH_LIVESHELL_INIT_TIMEOUT (default 300s) into the liveness-governed init backstop, preserving the knob from a superseded dev-side stopgap. Liveness still governs; this only tunes the wedged-but-alive backstop for a pathological host. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ell init Replaces the flat 5s init handshake deadline (which false-killed an alive-but- slow bash under load — reader threads starved while the marker sat unread) with a liveness-governed wait: succeed on marker sync, fail fast on real bash exit (naming rc), far 300s absolute backstop (env-overridable via METASMITH_LIVESHELL_INIT_TIMEOUT) only for a wedged-but-alive shell. The command/transform path (timeout=None) is unchanged. Real-process regression tests incl. an @slow GIL-starvation load repro. Supersedes the uncommitted 5s->30s dev stopgap (its env knob folded in). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…CACHEDIR + slurm array LiveShell init: liveness-governed handshake (activity-based timeout, env-overridable backstop). containers: honor APPTAINER_CACHEDIR for the apptainer image store. slurm.nf: default process array size 20 -> 100. Repo cleanup of vestigial files. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
phy0x1a79ed
changed the title
…fore dedup
When N distinct container subtypes (provides:bbtools/megahit/seqkit) each
structurally satisfy one generic `container` group_by requirement, the solver
correctly merges them (merged_endpoints), but the generator's collapse joined
their instances into given_map[canonical] WITHOUT normalizing dtype. Because
DataInstance.instance_id hashes path+dtype_name+parent_lib (NOT dtype), the
WithDType() retype copies dedup-collided with the un-retyped originals and were
discarded. group_by_instances kept N distinct dtype.key, so codegen emitted a
malformed o.group('<A>', [<B>], ...) -> NullPointerException in
Orchestrator.group at runtime, and only 1 of N containers pulled.
Fix: retype-before-dedup in the merged-endpoint collapse so the group_by
requirement sees one canonical dtype -> one input channel, N paths, homogeneous
fan-out. dtype_name is preserved so manifests still label each container by its
true subtype. Backstop: the silent plural-dtype Log.Warn is now a stage-time
ValueError so a bad o.group can never silently ship again.
Regression: tests/integration/test_multicontainer_groupby.py (stage-time, asserts
o.group by-key in using) + tests/e2e_virtual/test_virtual_multicontainer.py (plan
executes to N pulled). Reusable pull_container_transform() in mock_transforms.py.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ELEASE_PROTOCOL.md
phy0x1a79ed
changed the title
The image is tagged from the live source tree (-bd re-stamps build_hash.txt) but installs whatever sdist sits in dist/ (Dockerfile COPYs ./dist/*.tar.gz), and -bc packages that same sdist. An edit between -bp and -bd/-bc would ship an image tagged with a hash that doesn't match the code baked into it (and a mismatched conda build). Add _assert_dist_matches_source: recompute the live source hash and require dist/<name>-<ver>+<hash>.tar.gz, else fail with a 'rerun -bp' message. Override with MSM_SKIP_DIST_CHECK=1. Wire it into -bd and -bc; add a hermetic regression test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Under process.array=100 fan-out, every array task open()s the same shared control-plane YAMLs (task.yml, data-lib _metadata, agent.yml) through the apptainer /msm_home bind at once, and the bind/FUSE layer sheds reads with errno 108 (ESHUTDOWN). Tasks that exhaust tries are dropped silently (errorStrategy=ignore), losing a tool's output (amrfinder in the reported fir run wMxB557Y). T1: the host-side bootstrap now rsyncs the small control-plane subset (agent.yml, the task dir incl. transforms + _metadata images, each data lib's _metadata only) into the task's own $SLURM_TMPDIR scratch and threads a stage_root through execute_transform -> StageAndRunTransform so the loads resolve against the node-local copy (/ws/_metasmith/stage) instead of /msm_home. Reads the real $AGENT_HOME Lustre path, never the failing bind. Fail-open: no SLURM_TMPDIR / no rsync / copy error -> shared read as today, so the local executor path is byte-for-byte unchanged. Keys are location-independent, so a node-local copy loads to identical keys and cannot desync lineage (regression tests). T3: yaml_safe_load now retries OSError/errno-108 with its existing backoff (previously only empty parses), so any residual shared read survives a transient transport shutdown instead of failing the task. T2 (compute-node-local relay re-point) deferred pending the sockeye topology observation; the DIRECT branch already starts a --local node relay. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…an-out The per-task bootstrap bound the dev/metasmith overlay straight from Lustre; under SLURM array fan-out ~100 concurrent tasks read the same package files at import time and Lustre intermittently returned a partial readdir (errno 108), so a submodule (coms.ipc) momentarily vanished and the import crashed the task (exit 127), cascading to a missing relay binary and silent errorStrategy retries. Stage the overlay to per-task node-local scratch (same scheme as the existing control-plane staging; /ws is node-local under SLURM) and bind that instead; fail-open to the shared Lustre bind. Interactive msm launcher left unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…phantom guard Root cause (reproduced on fir Lustre, array=100): under SLURM array fan-out up to ~array-size tasks land on ONE node and each per-task rsync (ddd5130) walks the SAME shared dev-overlay tree at once. The metadata storm makes the Lustre client shed reads with errno 108 (ESHUTDOWN) and return a SILENTLY-INCOMPLETE copy, so a submodule (models.workflow) vanishes -> ModuleNotFoundError -> exit 127. Faithful repro: naive per-task rsync = 558 errno-108 events, 93/97 tasks incomplete; per-node-once = 0 / 0. Fix (agents.py bootstrap heredoc): - Dev overlay: per-node-once flock stage to /tmp/msm_devstage_$USER/$JOBID, keyed by the SLURM (array) job id (always this run's overlay, never stale on a reused node). Winner verifies models/workflow.py + coms present and file count >= 50 before publishing the stamp; retries transient errno-108 x3 with backoff; fails-open to the shared Lustre bind. Collapses ~N per-node reads to 1. - Control-plane subset: task-specific, so wrapped its rsyncs in retry-with-backoff x3, fail-open unchanged. Defense-in-depth for the distinct exit-126 SUBMITTED phantom (array child dies before writing .command.begin -> Nextflow hangs forever): inject_array_child_guard in models/paths.py installs an EXIT trap (needed under the dispatcher's set -e) into the array .command.sh that force-writes .command.begin + .exitcode iff the child did not, never clobbering a real exitcode. Wired into bin/sbatch fix_paths. Tests: tests/path_overhaul/test_sbatch_array_phantom_guard.py (5, incl functional bash -ue EXIT-trap), tests/integration/test_overlay_staging_logic.py (staging logic: verify/retry/flock/fail-open). Fast cohort green, no regressions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Complements the per-node-once overlay staging with two dynamic de-sync strategies in the same bootstrap heredoc (user-requested): - Adaptive start jitter: before the heavy Lustre reads, sleep random in [0, min(300, 3*SLURM_ARRAY_TASK_COUNT)]. Peak start rate ~1/3s; a 100-way array spreads over ~5 min (unnoticeable at that scale), a 3-way array over <=9s, a single/non-array task not at all (efficiency). Opt-out METASMITH_NO_START_JITTER=1. Also de-syncs the container-extract and control-plane reads that overlay staging does not cover. - Exponential backoff with full jitter: msm_backoff sleeps random(0, min(60, 2^attempt)) -- near-zero without contention, backs off dynamically on real errno-108. Replaces the linear sleeps in the dev-overlay and control-plane staging retry loops. Tests: tests/integration/test_start_jitter_backoff.py (+start_jitter_backoff.sh mirror): window scales & caps at 300, delay in [0,window], backoff exponential but bounded. Generated-shell integration still ALL GREEN. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Use the defensive ${VAR:-} form for METASMITH_NO_START_JITTER and
SLURM_ARRAY_TASK_COUNT in the fan-out start-jitter guard so an unset
variable can never abort the bootstrap under a nounset (set -u) shell
context. Production runs without set -u, but this makes the guard robust
regardless. Validated under real multi-node SLURM array fan-out on
sockeye (job 12083186): 100/100 tasks complete, 0 errno-108, per-node
overlay staged exactly once per node (6 reads / 6 nodes), jitter spread
1-296s across the 300s window.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rsync tree-walk The per-node-once overlay staging (commit 3adf4f0) collapsed N concurrent Lustre reads to 1 via an flock, but that one remaining read was still an `rsync -a` tree-walk of ~70 files -- hundreds of metadata ops, the exact op-class that triggers the errno-108 (ESHUTDOWN) client eviction under array fan-out -- and it was keyed by SLURM_ARRAY_JOB_ID, coupling staging to the scheduler. Deliver the overlay as a single tarball instead. Each node natively `cp`s it to node-local scratch, `tar -x` extracts it, verifies completeness, then binds it: * the per-node Lustre read is now ONE streaming file (what Lustre stays healthy under) instead of a readdir walk; the ~70 small-file writes land on node-local disk during extraction; * a truncated archive fails `tar -x` LOUDLY, killing the old silent-incomplete -copy (rsync exit 0 + missing submodule -> exit 127) failure mode; * the cache is keyed by the tarball's own `stat` (mtime+size) -- one metadata op, scheduler-agnostic, content-fresh (a reused node never serves a stale overlay; an identical tarball is reused for free). SLURM_ARRAY_JOB_ID is dropped entirely. The flock (per-node-once) and start-jitter/backoff are kept; all paths still fail-open to the shared Lustre tree bind, and the tree is still shipped as that fallback + the dev-run gate. - main/local_mock/rsync.py: build metasmith.tar (throwaway tempdir) + push it to $home/dev/metasmith.tar alongside the tree. - agents.py run_container: rewrite the staging block to cp+extract the tarball. - tests/integration/overlay_staging_logic.{sh,py}: rewrite to the tarball path (atomicity, cp+extract+verify, per-node-once, fail-open on a corrupt tarball, stat-key freshness). 7/7 green. Validated on sockeye under real multi-node SLURM array (job 12084073, array 0-99): 100/100 COMPLETED, 0 errno-108, bind=node_local 100/100, complete=yes 100/100 (all 73 files), per-node-once proven = 100 tasks across 5 nodes triggered exactly 5 cp reads (1/node, incl. a node packing 26 tasks), jitter spread 0-297s. See plans/03-tarball-dev-overlay.md and 02-errno108-overlay-fanout-rca.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rlay + per-node-once staging) Brings the SLURM array fan-out hardening stack onto dev: - errno-108 (ESHUTDOWN) -> exit-127 root fix: per-node-once overlay staging and the exit-126 SUBMITTED phantom guard (3adf4f0); - adaptive start-jitter + exponential backoff for the fan-out (bc9d044, 4dee272); - dev overlay now delivered as a single tarball (native cp + tar -x + bind), keyed by stat-of-tarball, dropping SLURM_ARRAY_JOB_ID (ad136d8); - concurrent-relay control-plane node-local staging + yaml safe-load retry (a7497c7). Validated e2e on sockeye under real multi-node SLURM arrays (jobs 12083186 rsync, 12084073 tarball): 100/100 complete, 0 errno-108, per-node-once proven. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Folds in the post-0.18.7 dev work: per-node-once overlay staging + phantom guard for the errno-108 array-fanout exit-127, tarball-based dev overlay delivery, adaptive bootstrap start-jitter + exponential backoff, control-plane node-local scratch staging under SLURM array fan-out, and the dev.sh dist/source-hash guard on -bd/-bc.
phy0x1a79ed
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Standing release PR from the fork's
releasebranch. Upstreamreleaseis at 0.18.3; this brings it to 0.18.8 (published artifacts below).Highlights by version
dev.shguard that blocks-bd/-bcwhendist/doesn't match the source hash.group_byinstances before dedup so N distinct container subtypes pull correctly); addRELEASE_PROTOCOL.md.APPTAINER_CACHEDIRfor the apptainer image store (single-pointContainer._store_root); SLURM array default size 20→100.dev.shfixes,examples/minimal library + HPC deploy smoke runner.Verification
Fast dependency-free test tier green: 345 passed, 1 skipped, 0 failed (
-m "not docker and not e2e_agentic and not nextflow and not slow and not network").Published artifacts (0.18.8+60556ca)
quay.io/hallamlab/metasmith:0.18.8-60556ca(+latest)anaconda.org/hallamlab/metasmith0.18.8v0.18.8A maintainer with upstream write access merges this.