Downstream fmriprep stages regenerate three anat outputs that `--derivatives` should reuse (upstream smriprep)
- Dominant language
- Python
- Stars
- 1
- Forks
- 4
- Avg merge
- 15h 39m
- Merged PRs (30d)
- 24
Description
con/mechababs#102 adds `--derivatives` to the downstream fmriprep stages so they reuse the anat stage's output instead of recomputing a divergent anatomical set (con/mechababs#78). It works: the divergent set collapses to a small residue. This issue tracks the residue, which is upstream.
Three files are still regenerated by the downstream stage even though they are present in the anat derivative:
| File | Upstream state |
|---|---|
| `sub-*_desc-ribbon_mask.nii.gz` | nipreps/smriprep#509, fixed by nipreps/smriprep#511 (merged 2025-11-18) — **unreleased** |
| `sub-*_desc-preproc_T2w.nii.gz` | nipreps/smriprep#510, fixed by nipreps/smriprep#512 (merged 2025-12-01) — **unreleased** |
| `sub-*_hemi-{L,R}_space-fsaverage_desc-reg_sphere.surf.gii` | nipreps/smriprep#575 — filed by us |
All three are the same shape of problem: smriprep's `collect_derivatives` does not populate the `precomputed` key that the anat workflow already checks before deciding to skip a stage. Nothing is wrong with our flags, and nothing in con/mechababs#102 needs to change.
The image that ran carries smriprep 0.19.2 (read from the SIF), which predates both merged fixes.
## Why this is not urgent for most datasets
We pass `--fs-subjects-dir` as well as `--derivatives`, so the regenerated files are produced deterministically from the reused FreeSurfer directory and come out data-identical to the anat stage's copies — independently verified by @lobennett on a second dataset (arrays match 3/3 subjects; two separate runs agreed exactly). A consumer reading either copy gets the same data.
## Where it does matter: T2w datasets
`desc-preproc_T2w` is the exception. It is not a deterministic format conversion — it re-runs the T2w-to-T1w coregistration and resample, so the two copies differ numerically (median absolute difference 0.01 to 0.19 against 99th-percentile intensities of ~1450 to 1950, measured by @lobennett). That reproduces con/mechababs#78's original problem exactly: two anat sets in one study that differ by construction, with no way for a consumer to tell which to read.
ds003097 has no T2w, so nothing we have run is affected. A T2w-bearing dataset processed before the upstream fix ships would be.
**Open decision:** accept-and-note, or hold T2w datasets until the fix lands. Leaning accept-and-note given the magnitude, but it should be a decision rather than an oversight.
## What has to happen before this is fixed for us
- [x] smriprep merges the fsaverage sphere fix (nipreps/smriprep#575)
- [ ] **smriprep cuts a release.** The step that has already failed twice — nipreps/smriprep#511 and nipreps/smriprep#512 have been merged since Nov/Dec 2025 and have reached no user. Latest release is 0.19.2 (2025-09-24); master is 122 commits ahead.
- [ ] fmriprep picks the new smriprep up in `pixi.lock` and releases. Mechanism unconfirmed: `pyproject.toml` declares `smriprep >= 0.19.2`, and `.github/workflows/pixi-lock.yml` relocks on PRs, but pixi will not necessarily upgrade a package whose locked version already satisfies the constraint — a deliberate bump may be needed. Watch whether the first fmriprep PR after a smriprep release moves that lock line.
- [ ] fmriprep publishes the docker image — automatic, `.github/workflows/docker.yml`
- [ ] ReproNim/containers mirrors it — automatic, no ask needed; `create_singularities-cron.sh` runs on a cron. Note this arrives as a *new* `bids-fmriprep--.sif`, not a rebuild of 25.2.5
- [ ] mechababs bumps its pipeline YAMLs to the new fmriprep version and re-vendors the container
- ~[ ] decide what happens to already-processed datasets (see the T2w note above)~
Only the last two are ours. The first three are the real latency and none is under our control.
Refs con/mechababs#78, con/mechababs#102.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the downstream --derivatives behavior described here, then inspect pyproject.toml, pixi.lock, .github/workflows/pixi-lock.yml, and .github/workflows/docker.yml. Track whether a smriprep release updates the lockfile and image, then check the pipeline YAMLs and vendored container. Done means the released upstream fixes are propagated and the three outputs are reused without divergent T2w results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- devops, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100