Config composition leaks: cluster/runtime settings land on the pipeline axis
- Dominant language
- Python
- Stars
- 1
- Forks
- 4
- Avg merge
- 15h 39m
- Merged PRs (30d)
- 24
Description
## Problem
A run composes three axes — dataset × pipeline × cluster — but today cluster- and container-runtime settings leak onto the **pipeline** axis, where they don't belong: SLURM partition, templateflow/FreeSurfer-license host binds, container flags (`--containall`, `--userns`, `$JOB_TMP:/tmp`). A pipeline YAML that hardcodes a site's paths/partition can't move to another cluster without editing it.
## Direction (not settled)
Treat the invocation as a **need↔provider join**, not a flat merge: the pipeline declares abstract *needs* (a mountpoint, a partition, writable `/tmp`) with no host paths; the cluster declares *providers* (the host path, the partition name); the composer joins them, and code branches on capability, never on cluster identity.
- Some of this is already babs-native (templateflow, fs-license) — we're double-handling.
- A generic host-file need/provider mechanism might be worth pushing to babs (an `EXTRA_FILES`-style config), complementing `PennLINC/babs#385` (`common_paths`) for files outside the sparse-checkout.
- `merge_config` composing cluster × pipeline (instead of overwriting) is the mechababs-side lever.
Fuzzy for now — the design is in progress; this issue is the tracker.
Contributor guide
No contributing guide indexed for this repository
Research direction
This is a design tracker with no files or tests named. Start by reading the need↔provider direction and locating the merge_config composition mentioned in the issue. Done means agreeing on and implementing a settled separation of pipeline needs from cluster providers, including the listed runtime settings, with the design no longer marked fuzzy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, devops, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100