Track intermittent CI test failures (code/CI/environment triage)

> [!IMPORTANT]
> **These started as triage candidates, not confirmed bugs.** A 30-day automated
> review surfaced recurring *intermittent* CI failures as a starting point for
> investigation. Each candidate could resolve to a **code** issue, a **test**
> issue, or a **CI/environment** issue. As candidates are root-caused, this
> issue is updated with the final disposition — see **Resolution status** below.
> Several "looked numerically unstable" candidates turned out to have concrete
> non-tolerance root causes; the term "flaky" is deliberately avoided for those.

> The environment is often the root cause (e.g. timeouts under Valgrind), but
> not always — see `ShapeLabelMapFixture`, which masqueraded as tolerance
> instability and was in fact a sign-arbitrary algorithmic result.

## Summary

A 30-day review (2026-05-25 → 06-25) of ITK CI failures across **GitHub Actions**
(`ITK.Pixi`, `ITK.Arm64`) and **CDash** (Azure DevOps + Kitware nightlies;
~18,920 failed-test rows, 676 distinct test names) found that most red CI is
**CI-environment/infrastructure**, with a small set of tests that fail
intermittently across *unrelated commits and multiple platforms*. This issue
tracks both so fixes/mitigations can be prioritized.

Methodology & how to read the tables

Failures were aggregated by test name × platform × commit/day. The signal that
separates a likely-recurring failure from a one-off breakage:
- **Cross-commit** (fails on several unrelated source revisions) and/or
- **Cross-platform + multi-day** (fails on Linux *and* Windows *and* macOS over
many distinct days).

A test failing many times on a **single day / single build** is almost always a
transient breakage that was then fixed — those are excluded below.

## Resolution status (updated as candidates are root-caused)

| Candidate | Final classification | Disposition |
|---|---|---|
| `ShapeLabelMapFixture` | **Two causes, both concrete (not tolerance instability).** `*_Direction` cases: OBB *origin* is a corner of a sign-arbitrary eigenvector frame → two equally-correct answers; latent defect disabled since 2017 (`0d9a5ed6`), re-exposed (not introduced) by the VNL→Eigen backend swap #6475. Non-`Direction` cases: genuine FP-model scalar precision, Windows-MSVC-only. | **In progress in #6521**: full-precision baselines for the scalar group + invariant-geometry assertions (frame-invariant OBB **center** + vertex-set membership) for the sign-arbitrary group. Origin corner to be documented non-portable. |
| `ParallelSparseFieldLevelSetRobustness.*` | **Real test/code issue, surfaced under Valgrind** (per @blowekamp: Valgrind defects reproduce locally; not a deadlock — #6519 closed after 770+ runs showed zero hangs). Earlier deadlock hypothesis withdrawn. | Open: address the Valgrind-reported defects; the 300 s watchdog self-aborts under CI load are a secondary CI-load symptom. |
| `itkComposeBigVectorImageFilterTest` | **CTest timeout under memory pressure** (100-channel `VectorImage`, no explicit `TIMEOUT`, `RESOURCE_LOCK MEMORY_SIZE`). | Open: set an explicit generous `TIMEOUT`. |
| `PythonLazyImportTime` | **Test-design assertion**: 5000 ms wall-clock ceiling (load-sensitive) + 750-module ceiling (deterministic). | Open: raise/relax the wall-clock ceiling or make it non-blocking on loaded runners. |
| `MaskedAssignFixture` | Assertion failure (`Completed (Failed)`); remote-module test (`*-Favorite-Remotes` nightly), not in core tree. | **Resolved**: addressed by @blowekamp in #6542. |
| `itkMeshFileReadWriteTest01` | Originally hypothesized as round-trip comparison (VTK ASCII FP formatting/precision). 2026-07-10 evidence points earlier in the test: two windows-2022 `Pixi-Cxx` failures in 0.04 s with no diagnostic output, on unrelated PRs (#6569, #6577); one passed on rerun of an identical tree. | Open: capture the test's stderr in the Pixi workflow to disambiguate startup/first-read failure from FP precision; still intermittent as of 2026-07-10. |
| `itkMCI_DSC_case_10_binary` | **New candidate 2026-07-10**: SEGFAULT on macos-15 `Pixi-Cxx` (PR #6576, diff unrelated to the MorphologicalContourInterpolation remote module); sibling `case_2` passed in 210 s on the same run. `RUNS_LONG` under a loaded runner fits the memory/load-pressure family, but one occurrence does not yet separate code from environment. | Open: track for cross-commit recurrence. |

## A. CI-environment / infrastructure failure modes (dominate red *runs*)

These are not test bugs; they prevent runs from producing a meaningful result.

| Mode | Observed | Likely layer |
|---|---|---|
| ExternalData download failures (CID/tarball fetch) | ~39% of red `ITK.Pixi` runs | CI/network/cache |
| Dashboard `exit 255` "warnings-as-fatal" (0 real test failures) | ~74% of red `ITK.Arm64` runs | CI scripting (`ci_completed_successfully`) |
| Runner setup / apt-disk / tool-setup (exit 127) | several `ITK.Pixi` runs | CI/runner |
| One Windows nightly config red for many days (`Windows11-VS22x64-RelWithDebInfo-Favorite-Remotes`) | ~32 tests × 15 days | single machine/config |

## B. Tests that fail intermittently across commits/platforms

These began as candidates; current understanding is in **Resolution status** above.
The original evidence is retained here for reference.

| Candidate | Evidence |
|---|---|
| `ShapeLabelMapFixture` (5 `3D_*` cases) | 187× / 30 days. Two modes: 3 non-`Direction` cases **Windows-MSVC-only** (105×); 2 `_Direction` cases **cross-platform** (Win+macOS+Linux). |
| `ParallelSparseFieldLevelSetRobustness.SweepRepeat` / `.ConcurrentMultiPipeline` | 132× / 102× over 18 days; Linux+Win+macOS incl. TSan/Valgrind. |
| `itkComposeBigVectorImageFilterTest` | 92× / 18 days; mostly coverage/memory-pressure builds. |
| `PythonLazyImportTime` | 115× / 18 days; Win+Linux Python. |
| `MaskedAssignFixture.SetGetPrint` | 276× / 18 days; concentrated in valgrind/coverage. |
| `itkMeshFileReadWriteTest01` | 4 unrelated commits; ubuntu+windows. +2 windows-2022 Pixi occurrences 2026-07-10 (see comment thread); rerun-green on identical tree. |

Explicitly excluded (single-day transients — NOT recurring)

`ComputeImageSpectralDensityTest` (06-12), `GeneralizedEigenDecomposition.IdentityPencilMatchesStandard`
(06-19), `itkImageFileWriterStreamingTest1_2/_3` (06-14), and a 120-test mass
red on one Windows nightly (06-25) — each a single bad build/commit that was
fixed, not recurring.

## Lesson learned

A test that flaps with the *texture* of numerical instability is not necessarily
a tolerance problem. When two backends can each return a different correct
answer (sign of an eigenvector, ordering of a degenerate decomposition), the
test is asserting an arbitrary representative — the fix is an **invariant
assertion**, not a looser bound. `ShapeLabelMapFixture` is the worked example:
every signal above the ITK test interface pointed at FP tolerance; the real
cause (algorithmic backend swap producing a sign-flipped but equally-correct
result) was only found by tracing below the test interface.

## Suggested next steps

- Treat **Section A** as infrastructure work (ExternalData fetch retry/caching;
reconsider warnings-as-fatal on ARM) — likely the biggest reliability payoff.
- Treat **Section B** per the **Resolution status** table; confirm/deny each
before any code change.

Happy to break any individual candidate into its own focused issue.