sharing steps across studies
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 161
- Forks
- 49
- PR merge metrics
- No merged PRs in 30d
Description
It would be great to be able to share some steps across studies, e.g. some pre-processing
example
```yaml
study:
- name: step1
description: preproc
run:
cmd: |
echo "Some preprocesisng code here, possibly long
procs: 1
nodes: 1
walltime: "00:01:00"
- name: step2
description: some code
run:
cmd: |
echo "Actual work"
$(LAUNCHER) $(CODE) -in $(MODEL1)
procs: 1
nodes: 1
walltime: "00:01:00"
```
assuming step1 is shared by many maeastro studies it could become something like this:
```yaml
study:
- name: step1
path: $(SHARED_MAESTRO_FILES)/preprocessing.yml
step: standard_preproc
- name: step2
description: some code
run:
cmd: |
echo "Actual work"
$(LAUNCHER) $(CODE) -in $(MODEL1)
procs: 1
nodes: 1
walltime: "00:01:00"
depends: step1
```
And the preprocessing.yml file could look like this
```yaml
study:
- name: standard_preproc
description: standard preprocessing
run:
cmd: |
echo "Some preprocessing code here, possibly long
procs: 1
nodes: 1
walltime: "00:01:00"
- name: advanced_preproc
description: a more advanced preprocessing step
run:
cmd: |
echo "Some preprocessing code here, possibly long
procs: 1
nodes: 1
walltime: "00:01:00"
```
That way we can share and update the preprocessing step across many studies. I do realize this is not great for provenance, but maestro could build a "Final" yml file with the pieces brought over from all the external yml files.
The concept could also be exported to resources where a resources.yml could exit in a standard place but with diffierent values depending on the machine you're running on.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the proposed preprocessing.yml and resources.yml structures and tracing how study YAML files are currently loaded. Define how external steps, dependencies, resource values, and the generated final YAML should behave. Done means shared steps can be referenced by multiple studies and the assembled configuration preserves provenance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100