Chained stage: run our selection to trim upstream-only wasted jobs
- Dominant language
- Python
- Stars
- 1
- Forks
- 4
- Avg merge
- 15h 39m
- Merged PRs (30d)
- 24
Description
A chained downstream stage (fmriprep minimal/full/resampling) passes **no** inclusion to `babs init`, letting babs derive the job set by intersecting its inputs (raw BIDS ∩ each upstream output). That correctly drops subjects an upstream never produced (e.g. anat that failed).
But babs intersects on subject **presence**, not datatype. So a subject the upstream *did* produce but that the downstream's own `selection:` rule would exclude — e.g. an anat-only subject (T1w, no BOLD) that got an anat derivative — still gets a downstream job, which fails fast (no BOLD for minimal). **Wasteful, not incorrect** (the job fails, no bad output).
**Optimization:** for a chained cell, run our own `selection:` rule and pass the intersected list (`select(rule) ∩ upstream-produced`) as the inclusion, trimming those wasted jobs. Requires reading the upstream's produced-subject set (e.g. from its merged output / `processing_inclusion.csv`).
Deferred from the chained-inputs work (#72). The current fail-fast behavior is acceptable for the shakeout; this is throughput/cleanup. The code it describes lands with #77 — the site is the inclusion `TODO` in `iterate.py` `scaffold`.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the inclusion TODO in iterate.py's scaffold, then review the chained-inputs work from #72 and the code landing with #77. Trace how the upstream merged output or processing_inclusion.csv exposes produced subjects and how the selection rule is evaluated. Done means a chained cell passes the selected-and-produced subject list to babs init, avoiding jobs excluded by selection.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 56/100