fmriprep `DatasetLinks.raw` carries the per-job scratch abspath; should be the relative `sourcedata/raw`
- Dominant language
- Python
- Stars
- 1
- Forks
- 4
- Avg merge
- 15h 39m
- Merged PRs (30d)
- 24
Description
## What
fmriprep's `dataset_description.json` in a babs-produced derivative carries the job's scratch abspath in `DatasetLinks.raw`:
```json
"DatasetLinks": {
"raw": "/scratch4/workspace/f006rq8_dartmouth_edu-mechababs/job-61975422-3-sub-CN003/ds/sourcedata/raw",
"templateflow": "https://github.com/templateflow/templateflow"
}
```
Seen on ds003643 `fMRIPrep-25.2.5+minimal` (Unity, 2026-07). Every subject's zip carries its own value (`job-…-1-sub-CN001`, `-2-sub-CN002`, `-3-sub-CN003`), and after `add-archive-content --existing overwrite` the derivative keeps whichever subject was extracted last, pointing at a directory that no longer exists.
## Why it matters
- **M2 litmus:** a published derivative whose `dataset_description.json` names a dead cluster path is not portable, and the link is wrong for every subject but one.
- **Optional zipping:** with zip off, every job branch adds the same `dataset_description.json`, and babs's octopus merge dies on the first non-identical copy. This file is the one that differs; `.bidsignore`, `desc-*_dseg.tsv` and `logs/CITATION.md` are byte-identical across subjects (verified on the three ds003643 zips). cf PennLINC/babs#327.
- Surfaced while testing datalad/datalad#7920, whose new `--overwrite-prior-check` flagged exactly this file.
## Where it comes from
babs's job template passes the input as an absolute path, `"${PWD}/{{ bids_app_input_dir }}"` (`babs/templates/bidsapp_run.sh.jinja2`), with `PWD` the per-job scratch clone. fmriprep writes the bids dir it was given, resolved to absolute, as `DatasetLinks.raw` (`fmriprep/utils/bids.py`, `write_derivative_description`; introduced by nipreps/fmriprep#3252). Passing a relative path changes nothing, since fmriprep resolves it.
## The right value
Per the BIDS spec, a scheme-less `DatasetLinks` value resolves relative to the dataset root. The merged derivative carries the raw data as the `sourcedata/raw` submodule, so the correct value is the relative `sourcedata/raw`: byte-identical across subjects, valid after merge and extraction, and portable.
Only babs knows both the invocation and where raw lands in the merged derivative, so the fix is upstream in babs, either:
1. rewrite `DatasetLinks.raw` to `sourcedata/raw` in the app's output after the app runs (the only route to the correct value), or
2. bind the job dir at a stable container path instead of `${PWD}` (same move as #6 proposes for the recorded run command), which makes the value identical across jobs and unblocks the octopus merge, but leaves a container-internal path in a published file.
fmriprep-side, a way to set `DatasetLinks` explicitly would let babs do this without editing the app's output; not filed there.
## Tracking
`babs-upstream`, not yet filed in `PennLINC/babs`; drop `upstream-NOT-FILED` and add the `PennLINC/babs#N` here once it is. cf #6 (abspaths in the run record, same bind-mount option), PennLINC/babs#327 (zip-off merge).
---
*Generated by Claude Code.*
Contributor guide
No contributing guide indexed for this repository
Research direction
Read babs/templates/bidsapp_run.sh.jinja2 and the fmriprep call described in fmriprep/utils/bids.py, then inspect how generated dataset_description.json files are collected and merged. Confirm the chosen upstream approach preserves a relative sourcedata/raw link and makes subject outputs byte-identical and portable after merging.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100