con / con/mechababs

Config composition leaks: cluster/runtime settings land on the pipeline axis

Open
#97 1 comment 0 reactions 0 assignees View on GitHub
automation decision fuzzy/slop generalize
Dominant language
Python
Stars
1
Forks
4
Avg merge
15h 39m
Merged PRs (30d)
24

Description

## Problem

A run composes three axes — dataset × pipeline × cluster — but today cluster- and container-runtime settings leak onto the **pipeline** axis, where they don't belong: SLURM partition, templateflow/FreeSurfer-license host binds, container flags (`--containall`, `--userns`, `$JOB_TMP:/tmp`). A pipeline YAML that hardcodes a site's paths/partition can't move to another cluster without editing it.

## Direction (not settled)

Treat the invocation as a **need↔provider join**, not a flat merge: the pipeline declares abstract *needs* (a mountpoint, a partition, writable `/tmp`) with no host paths; the cluster declares *providers* (the host path, the partition name); the composer joins them, and code branches on capability, never on cluster identity.

- Some of this is already babs-native (templateflow, fs-license) — we're double-handling.
- A generic host-file need/provider mechanism might be worth pushing to babs (an `EXTRA_FILES`-style config), complementing `PennLINC/babs#385` (`common_paths`) for files outside the sparse-checkout.
- `merge_config` composing cluster × pipeline (instead of overwriting) is the mechababs-side lever.

Fuzzy for now — the design is in progress; this issue is the tracker.

Contributor guide

No contributing guide indexed for this repository

Research direction

This is a design tracker with no files or tests named. Start by reading the need↔provider direction and locating the merge_config composition mentioned in the issue. Done means agreeing on and implementing a settled separation of pipeline needs from cluster providers, including the listed runtime settings, with the design no longer marked fuzzy.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.