con / con/mechababs

Cross-session datasets: anat and func in different sessions defeat session-level selection and processing (ds001499, ds001506, ds004496, ds003020)

Open
#156 2 comments 0 reactions 0 assignees View on GitHub
dataset decision pipeline:fmriprep
Dominant language
Python
Stars
1
Forks
4
Avg merge
15h 39m
Merged PRs (30d)
24

Description

Four priority datasets store the anatomical in one session and the functional runs in others. `add-dataset` derives `processing_level: session` for any dataset whose metadata rows name sessions, and everything downstream then works per (subject, session): the selection rule, the inclusion list, babs's sparse checkout, and the job. No single session holds both an anat and a BOLD, so these datasets cannot be processed at session level.

Two failure modes, both seen:

- **Fully cross-session** (ds001499, ds001506, ds004496): the fmriprep selection rule (`require_datatypes: [anat, func]`, `require_positive: [t1w_num, bold_num]`) matches no (subject, session) row, and the cell fails at scaffold with "no eligible subjects". MRIQC's rule is per-modality and passes. Skipped in the 2026-09 sweep (#86).
- **Partially cross-session** (ds003020): `sub-UTS02 ses-1` is anat-only, `ses-2..20` are func-only; `sub-UTS03 ses-1` has localizers so it passes. The 2026-08 run processed `ses-1` only, so the experiment's functional sessions were never touched, and the chained stages (`minimal`, `full`) inherited `(sub, ses-1)` from `anat` and ran on a session with no BOLD. The loud symptom was one failed job; the quiet one was a whole dataset under-processed.

The metadata is correct in every case (checked against `sourcedata+subjects+sessions.tsv`); this is a mechababs design gap, not an upstream data problem.

| dataset | subjects | sessions | shape |
|---|---|---|---|
| ds001499 | 4 | 58 | anat only in `ses-16` / `ses-10`; also #49 (a T1w that fails hash verification) |
| ds001506 | 3 | 77 | T1w only in `ses-anatomy`; func sessions list `anat` with `t1w_num = 0` |
| ds004496 | 30 | 115 | no session with both |
| ds003020 | 9 | 92 | anat in `ses-1` only; `UTS03` `ses-1` also has localizer BOLD |

### The decision

How a subject's anat session gets paired with its func sessions. Options, from least to most machinery:

1. **A per-dataset `--processing-level` override on `add-dataset`.** The operator says `subject` for these four; fMRIPrep collects the T1w from any session of the subject and processes all its BOLD runs, and babs runs the whole subject in one job. The escape hatch that unblocks them now. Cost: a subject with dozens of sessions is the #44 shape (one job, days of walltime); ds003020's 20 sessions per subject is that shape.
2. **Derive it.** `add-dataset` keeps `session` only when at least one session per subject (or per dataset) holds both anat and func, and otherwise derives `subject`. Same result as 1 without the operator having to know; the mixed case (ds000113, #14) needs the same check from the other side.
3. **Session pairing in selection:** an anat-session column on the inclusion list, so a func session runs with its subject's anat session staged beside it. Keeps session-sized jobs, but babs's sparse checkout is per session and nothing in babs or fMRIPrep expects a second session's anat, so this is upstream work.

Proposal: 1 now, 2 as the heuristic once 1 has run these four, 3 only if subject-level jobs prove too big for a dataset we need.

Related: #14 (the mirror case: the heuristic says `session` and babs refuses the one session-less subject), #44 (subject-level job size), #19 (subdataset-per-subject for very large datasets), #92 (babs joins inputs on subject presence, not datatype).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the add-dataset entry point where processing_level is derived, then trace the selection rule, inclusion list, sparse checkout, and job inputs described in the issue. Review related issues #14, #44, #19, and #92 before choosing the processing-level behavior; done means the four listed datasets can be processed without session-level under-selection or invalid anat/func pairing.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.