llnl / llnl/maestrowf

Add `branch:` key to dynamically generate workflow steps

Open
#455 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
161
Forks
49
PR merge metrics
No merged PRs in 30d

Description

**Disclaimer/Intro:**
Maestro is powerful, but does not support dynamic workflows like optimization. @jsemler has written a private package called `encorewf` that supports limited iteration. I asked ChatGPT o4-mini for a more general solution, and they (gender-neutral?) suggested the following. [My full chat with ChatGPT](https://chatgpt.com/share/6842d963-4cfc-800b-af10-1563596cda15) is [here](https://chatgpt.com/share/6842d963-4cfc-800b-af10-1563596cda15).

@jwhite242, @jsemler, @FrankD412, @daub1, @antimatterhorn, Please let me know what you think!

**Problem:**
Maestro’s current study spec requires all steps to be defined statically (or via parameter expansion). There is no built-in way to run a script at runtime that emits new step definitions based on completed tasks.

**Proposal:**
Introduce a new `branch:` field under `run:` that executes a user‐supplied command after its `depends:` finish. That command should print valid YAML step definitions to stdout, which Maestro will parse and inject into the DAG on the fly.

**Example:**
```yaml
- name: run-lulesh
run:
cmd: |
$(LULESH)/lulesh2.0 -s $(SIZE) -i $(ITERATIONS) -p > $(outfile)
depends: [make-lulesh]

- name: branch-postprocess
run:
# After all run-lulesh_* instances complete, run this script.
branch: |
python scripts/generate_postprocess_steps.py --input-dir $(OUTPUT_PATH)
depends: [run-lulesh_*]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the proposed scripts/generate_postprocess_steps.py example and the run/depends workflow entry points. Trace how Maestro validates static YAML and builds the DAG, then determine how generated step definitions could be validated and injected after dependencies finish. Done means branch-generated steps execute with the expected dependency behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, yaml
Domain
tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.