ds005747 / ds004044 MRIQC: OOM-killed at synthstrip even at 24G on a hard-enforcing cluster

MRIQC jobs are OOM-killed at the `synthstrip` node on a cluster that hard-enforces the memory cgroup, even after raising the SLURM grant from 16G to 24G. Observed on ds005747 and ds004044 (Unity, 2026-07-19); exit 137, with SLURM logging an explicit `Detected N oom_kill event`.

## Why

`synthstrip` loads a torch model whose RSS spikes. The app-side `--mem_gb` is a **soft nipype scheduling hint** (it caps how many nodes run concurrently), not a hard bound on any single subprocess — so the spike can push total RSS past the cgroup cap. The pipeline configs originally granted SLURM exactly the app's own target (16G/16G), i.e. zero headroom; that has since been raised to 24G, which is better but still not sufficient.

**Failure is subject-size-dependent, not random.** On ds004044, sub-01 passed at 24G while sub-02 — more BOLD, so heavier funcMRIQC — was killed. A retry at 40G passed.

## Cluster dependence

Cgroup memory enforcement is a **cluster property**. The same 16G config ran green on Discovery, which appears to enforce softly; Unity hard-enforces and kills. So an identical, pinned config is safe on one cluster and fatal on another — which also means a green run elsewhere is not evidence the headroom is right.

## State

ds005747/MRIQC is **deliberately left failed** — it was scanceled and kept as the worked example for the config-change recovery path (bump the pipeline's mem, `retire-derivative`, `iterate` re-scaffolds at the new grant, no hand-patched job script).

## Open — how to fix it properly

- **Raise the SLURM grant again** (24G → 32G or more). Simple; guesses again unless informed by a measurement.
- **Reduce parallelism** — lower the app-side `--mem_gb`/`--n_cpus` so nipype schedules fewer concurrent heavy nodes, trading wall-clock for peak RSS.
- **Right-size from data** — read `MaxRSS` from a *passing* job (`seff`/`sacct`); an OOM-killed job's `MaxRSS` is capped at the limit and therefore useless. This is the input the other two options need, and it feeds #3 (automate resource estimation).

Note also that duct's sampled `peak_rss` missed this spike on the earlier Discovery run (3.85 GB reported for a job whose real peak tipped a 16G cap), which is why "allocations left as-is" looked safe at the time — a documented instance of interval sampling missing a short allocation spike.

cf #3, #34.

Start at the pipeline configuration for the synthstrip node and review how --mem_gb and --n_cpus affect nipype scheduling versus the SLURM grant. Compare MaxRSS from a passing job using seff or sacct, then choose and validate a resource or parallelism change; done means the affected datasets complete under Unity's hard memory cgroup without OOM events.

Written by the indexing model from the issue text.