llnl / llnl/maestrowf

[Proposal] YAML Specification Improvements

Open
#147 5 comments 0 reactions 1 assignee Claimed by @FrankD412 View on GitHub
enhancement In Progress
Dominant language
Python
Stars
161
Forks
49
PR merge metrics
No merged PRs in 30d

Description

So I've been looking at some of the limitations of the YAML specification as it is, and I've come up with a new structure for it.

```
name: step_name
description: Some human readable English here for the step.
cmd: |
exec some commands here
restart: |
exec a restart here
post: |
exec a small-ish local step after the main step terminates
resources:
sched:
nodes:
tasks:
procs:
gpus:
cores per task:
```

The idea behind splitting out the resources is two fold:
1. This organization makes it so that Maestro can have monikers like `$(resource.)` and so long as the key exists in `resources:` it can be used. Users could then "manually" select which MPI they want to use in a fashion similar to `mpirun -n $(resources.tasks) ... some --command --to run`. Principally, user could even do shell math to computer MPI related values (and Maestro could even calculate high level settings inferring from statically set values).
2. Consolidation of resource settings and the ability to allow for more flexible step-wise specification of required resources. Sensible defaults could be provided, and a higher level batch could be specified, and then override values with step-wise values. It opens up a clear and simple line of application of batch parameters that can be consolidated in a uniform fashion while still giving the user more control.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.