llnl / llnl/maestrowf

Manually restarting an experiment after job completion

Open
#449 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
161
Forks
49
PR merge metrics
No merged PRs in 30d

Description

Suppose I run an experiment that dispatches N jobs, each restarting as many times as specified in restart_limit and restarting using code written in the "restart" section of the YAML script. I would like to be able to also restart the experiment at any point in the future, as opposed to the automatic restart. This way, I can manually restart later on if all of the original restarts fail or if I want to run additional times beyond the specified restart_limit. The restarts would occur within the same experiment folder, as opposed to generating a new folder which occurs when running "maestro run experiment.yaml". Does such a feature exist?

Contributor guide

Open the contributing guide

Research direction

Start by tracing the existing `maestro run experiment.yaml` flow and how the YAML `restart` section and `restart_limit` are handled. Determine how a later manual restart could reuse the existing experiment folder rather than create a new one, and define completion around supporting that workflow after the original jobs finish.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.