Capture mount dumps and preserve logs when a functional test fails#2062
Open
tyrielv wants to merge 3 commits into
Open
Capture mount dumps and preserve logs when a functional test fails#2062tyrielv wants to merge 3 commits into
tyrielv wants to merge 3 commits into
Conversation
When a functional test fails because its GVFS.Mount is unreachable (a hang or silent exit), we currently have no post-mortem data. Per-test enlistment .gvfs/logs are deleted at teardown and only survive as full-file contents inlined into the console by TestResultsHelper.OutputGVFSLogs -- lossy under parallel fixtures and empty when the mount hung. There is no process dump, so a mount deadlock leaves zero diagnostic signal. On test failure only, into a CI-uploadable diagnostics directory: - Capture a full-memory minidump of each still-running GVFS.Mount process for the failing enlistment (a hung mount is still alive at teardown). New Tools/MiniDump.cs P/Invokes MiniDumpWriteDump (dbghelp); Windows-only, best-effort, never throws. Full memory is required so managed call stacks are resolvable in WinDbg/SOS. - Robustly copy the enlistment .gvfs/logs. A plain File.Copy is tried first; if the file is locked (a hung mount still holds its log open), fall back to a read-only shared handle (FileShare.ReadWrite | Delete) and copy whatever has been flushed -- partial content is still useful. GVFSFunctionalTestEnlistment.CaptureFailureDiagnostics runs first in DeleteEnlistment, gated on TestStatus.Failed, before the enlistment directory is deleted. The mount-process PID discovery is extracted from KillMountProcess into a shared GetMountProcessIds helper. CI: functional-tests.yaml sets GVFS_TEST_DIAGNOSTICS_DIR and uploads it as a FailureDiagnostics artifact with if: always(). Assisted-by: Claude Opus 4.8 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
|
|
||
| public void DeleteEnlistment() | ||
| { | ||
| this.CaptureFailureDiagnostics(); |
There was a problem hiding this comment.
Mount dump capture runs after the mount has already been unmounted or killed.
| try | ||
| { | ||
| System.Diagnostics.Process.GetProcessById(pid)?.Kill(); | ||
| } |
There was a problem hiding this comment.
The PowerShell helper can still hang despite the later timeout.
GetMountProcessIds doubled the backslashes in the enlistment path before using
it in a PowerShell -like wildcard. In -like, '\' is a literal (not an escape),
so the doubled pattern never matched a real single-backslash command line:
CaptureFailureDiagnostics found no live mount and wrote no minidump, and
KillMountProcess (from which this was extracted) silently killed nothing.
Match on the enlistment's unique leaf folder id instead. It is present on the
GVFS.Mount command line (launched with PrimaryEnlistmentRoot), is unique, and
contains no path separators or wildcard metacharacters, so it needs no escaping.
Review follow-ups:
- The mount minidump is now captured before UnmountAndDeleteAll unmounts or
kills the mount process, instead of afterward from DeleteEnlistment. The
dump capture ran too late: by the time DeleteEnlistment ran, the mount had
already exited (cleanly, or via KillMountProcess after an unmount timeout),
so there was nothing left to dump. CaptureFailureDiagnostics now only
writes the mount dump; log preservation moved to a new CaptureFailureLogs,
still called from DeleteEnlistment after the mount is gone.
- GetMountProcessIds read the PowerShell helper's stdout with a blocking
ReadToEnd() before calling WaitForExit(10000). ReadToEnd() blocks until the
process closes its output handle, so if the helper itself hung, the later
WaitForExit timeout was never reached. It now reads output asynchronously
via OutputDataReceived/BeginOutputReadLine, so WaitForExit(10000) is the
only blocking call and actually enforces the timeout, killing the helper
if it does not exit in time.
Assisted-by: Claude Sonnet 5
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
tyrielv
force-pushed
the
tyrielv/ft-failure-diagnostics
branch
from
ae9be31 to
aa4fdf0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Capture mount dumps and preserve logs when a functional test fails
When a functional test fails in CI, the only diagnostic we get today is whatever
TestResultsHelper.OutputGVFSLogshappens to inline to the console. If aGVFS.Mountprocess is hung (the common flaky-failure signature — a test timesout waiting on the mount named pipe with no crash trace), there is no way to see
why after the fact: the process is killed during teardown and its state is lost.
This adds first-class failure diagnostics for the functional tests:
GVFSFunctionalTestEnlistment.CaptureFailureDiagnostics()finds the liveGVFS.Mountprocess(es) for the enlistment and writes a full-memory minidumpvia
MiniDumpWriteDump(newMiniDumphelper, P/Invoke intodbghelp). Thisis exactly what's needed to root-cause a hang.
.gvfs/logsare copied out witha fallback that reopens locked/partially-flushed files with a shared
read handle (
FileShare.ReadWrite | Delete), so we still capture logs that alive process is holding open.
functional-tests.yamlpoints diagnostics at a fixeddirectory (
GVFS_TEST_DIAGNOSTICS_DIR) and uploads it asFailureDiagnostics_<matrix>withif: always().Everything is best-effort and never throws, so it cannot turn a passing test red
or mask the real failure.
Incidental fix
GetMountProcessIds(extracted from the existingKillMountProcess) matched themount command line with a PowerShell
-likewildcard built from the enlistmentpath with its backslashes doubled. In
-like,\is a literal, not an escape,so the doubled pattern never matched a real single-backslash command line — it
found zero PIDs. This was a pre-existing latent bug:
KillMountProcesshas beensilently killing nothing. Now matches on the enlistment's unique leaf id, which
is on the mount command line and needs no escaping.
Validation
Exercised end-to-end in CI via a throwaway harness branch (a smoke test that
Assert.Fail()s with a live mount). TheFailureDiagnosticsartifact came backcontaining
GVFS.Mount_<pid>.dmp(~125 MB full-memory dump) plus the preservedclone/mount/prefetch logs.