COMP: Right-size CTest timeouts — Debug vs optimized defaults, multi-config Debug legs, and outlier tests

ITK sets no `DART_TESTING_TIMEOUT`, so every test inherits CTest's 1500 s default — roughly 40x what a normal test needs, meaning a hung test burns up to 25 minutes of CI before it is killed. A build-type-aware default plus explicit timeouts on a short list of genuine outliers would cut hang detection to minutes.

Timing evidence (CDash snapshot 2026-05-17 .. 2026-05-23 — not re-verified since)

Sample: **275,956 passing-test timing records** from **69 full-test builds** across 10 platform families (Linux/Windows/macOS x Debug/Release x TBB/noTBB, plus Python-wrapping and Continuous-main), collected from open.cdash.org project `Insight` via the GraphQL `build(id){tests{edges{node{name status runningTime}}}}` endpoint. Instrumented (coverage/valgrind/asan) builds excluded.

Pooled percentiles, seconds:

| p80 | p90 | p95 | p97 | p99 | p99.9 | p100 | mean |
|----|----|----|----|----|----|----|----|
| 0.36 | 0.97 | 2.31 | 4.26 | 13.86 | 162.2 | 1403.5 | 1.10 |

By build type — Debug dominates the tail:

| build type | n | p99 | p99.9 | max |
|---|---|---|---|---|
| Release | 139,493 | 7.53 | 35.69 | 255.9 |
| Debug | 99,800 | 37.68 | 262.85 | 1403.5 |

By OS max: Linux 269.7 s, macOS 625.1 s, Windows 1403.5 s (all maxima on Windows-Debug-TBB).

Cost of each candidate ceiling C (global default = 2C):

| C | default 2C | distinct tests needing an explicit TIMEOUT |
|---|---|---|
| 60 | 120 | 92 |
| 120 | 240 | 55 |
| 180 | 360 | 36 |
| 240 | 480 | 21 |
| 300 | 600 | 14 |

Outlier tests (same snapshot; max observed passing time > 300 s)

| test | max (s) | platform of max |
|---|---|---|
| itkMultiLevelSetsv4MalcolmImageSubset2DTest | 1403.5 | Windows-Debug-TBB |
| itkTestTransformGetInverse | 1356.6 | Windows-Debug-TBB |
| itkImageToHistogramFilterTest | 823.7 | Windows-Debug-TBB |
| itkLargeMetaImageWriteReadTest3 | 761.7 | Windows-Debug-TBB |
| itkDiscreteHessianGaussianImageFunctionTest | 694.3 | Windows-Debug-TBB |
| itkTimeVaryingBSplineVelocityFieldPointSetRegistrationTest | 542.3 | Windows-Debug-TBB |
| RelabelComponentImageFilter.BigRandom | 482.9 | Windows-Debug-TBB |
| MultiStageImageRegistration1Test | 402.3 | Windows-Debug-TBB |
| itkMultiLevelSetsv4WhitakerImageSubset2DTest | 400.9 | Windows-Debug-TBB |
| ImageRegistration15Test | 372.8 | Windows-Debug-TBB |
| itkVNLRoundProfileTest1 | 343.9 | Windows-Debug-TBB |
| itkDiffeomorphicDemonsRegistrationFilterTest02 | 329.2 | Windows-Debug-TBB |
| itkLargeMetaImageWriteReadTest4 | 325.4 | Windows-Debug-TBB |
| itkLargeMetaImageWriteReadTest2 | 318.8 | Windows-Debug-TBB |

Suggested explicit `TIMEOUT` ≈ 2x observed max, rounded up. `itkMultiLevelSetsv4MalcolmImageSubset2DTest` and `itkTestTransformGetInverse` already carry a 3600 s property.

Remote-module outliers (not in core, fix in their own repos): `itkMCI_DSC_case_2_binary` (~172 s) and `itkMCI_DSC_case_10_binary` (~136 s) in MorphologicalContourInterpolation (pending ingest, see #6209). Montage: `itkMontageSingleLayer3D` peaked at ~133 s.

Proposed work

1. Set a build-type-aware default in the top-level `CMakeLists.txt` **before** `include(CTest)` (currently line 170, and no `DART_TESTING_TIMEOUT` is set today): ~600 s for Debug, ~120 s for optimized. 600 s = 2x the ~300 s normal-test ceiling and keeps the explicit-override list to 14 tests; a much tighter default (e.g. 2x p99 ≈ 30 s) would require overrides on 150-270 tests and is unmaintainable.
2. Multi-config generators (Windows VS, Xcode) have no configure-time `CMAKE_BUILD_TYPE`, so the conditional cannot fire — set `CTEST_TEST_TIMEOUT` / pass `ctest --timeout` per config in the dashboard/CI scripts so the Debug legs get the Debug allowance.
3. Add explicit `set_tests_properties( PROPERTIES TIMEOUT <2x max>)` for the outliers above.
4. Investigate the Windows-Debug-TBB extremes — `itkMultiLevelSetsv4MalcolmImageSubset2DTest` ~1404 s and `itkTestTransformGetInverse` ~1357 s are ~5x the next platform. This looks like a Debug+TBB performance pathology worth fixing rather than merely tolerating.
5. Consider labelling the heavy tests `RUNS_LONG` consistently and excluding them from fast PR-gating legs, running them only on nightlies.
6. Re-derive the percentiles periodically as the suite evolves; the numbers above are a point-in-time snapshot.

Caveats

- Times are wall-clock on shared CI runners; absolute values vary with hardware load. The 2x margin absorbs this.
- The sample is 69 of ~250 daily builds — representative breadth, not exhaustive.
- **The percentiles and per-test maxima above are from a 2026-05-17..2026-05-23 CDash snapshot and have not been re-derived since.** Re-collect before acting on any specific number.

Related

- #6518 — Track intermittent CI test failures (flake-adjacent; this issue is about timeout *policy*, not individual flaky tests).
- #6209 — MorphologicalContourInterpolation ingest, where the MCI outlier timeouts belong.
- A prototype exists on a personal-fork branch `comp/test-timeout-debug-vs-release` (not upstream) implementing the 600/120 split plus explicit timeouts for `itkImageToHistogramFilterTest`, `itkDiscreteHessianGaussianImageFunctionTest`, `itkLargeMetaImageWriteReadTest1/2/3`, `itkMontageSingleLayer3D`.

---

Re-collect the CDash timing data first, since the listed percentiles and maxima are unverified. Read top-level CMakeLists.txt around line 170 and inspect the dashboard/CI scripts for multi-config timeout handling, then review the named outlier tests and their existing TIMEOUT properties. Done means validated defaults for Debug and optimized builds, appropriate multi-config behavior, and explicit timeout coverage for agreed core outliers.

Written by the indexing model from the issue text.