con / con/mechababs

Chained stage: run our selection to trim upstream-only wasted jobs

Open
#92 0 comments 0 reactions 0 assignees View on GitHub
automation
Dominant language
Python
Stars
1
Forks
4
Avg merge
15h 39m
Merged PRs (30d)
24

Description

A chained downstream stage (fmriprep minimal/full/resampling) passes **no** inclusion to `babs init`, letting babs derive the job set by intersecting its inputs (raw BIDS ∩ each upstream output). That correctly drops subjects an upstream never produced (e.g. anat that failed).

But babs intersects on subject **presence**, not datatype. So a subject the upstream *did* produce but that the downstream's own `selection:` rule would exclude — e.g. an anat-only subject (T1w, no BOLD) that got an anat derivative — still gets a downstream job, which fails fast (no BOLD for minimal). **Wasteful, not incorrect** (the job fails, no bad output).

**Optimization:** for a chained cell, run our own `selection:` rule and pass the intersected list (`select(rule) ∩ upstream-produced`) as the inclusion, trimming those wasted jobs. Requires reading the upstream's produced-subject set (e.g. from its merged output / `processing_inclusion.csv`).

Deferred from the chained-inputs work (#72). The current fail-fast behavior is acceptable for the shakeout; this is throughput/cleanup. The code it describes lands with #77 — the site is the inclusion `TODO` in `iterate.py` `scaffold`.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the inclusion TODO in iterate.py's scaffold, then review the chained-inputs work from #72 and the code landing with #77. Trace how the upstream merged output or processing_inclusion.csv exposes produced subjects and how the selection rule is evaluated. Done means a chained cell passes the selected-and-produced subject list to babs init, avoiding jobs excluded by selection.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
56/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.