llnl / llnl/magpie

Enhancement proposal: making timeout behavior more dynamic

Open
#9 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Shell
Stars
198
Forks
51
PR merge metrics
No merged PRs in 30d

Description

While testing, I found that when the setup time was shorter than expected, the job was unable to use all the walltime available because of the shutdown timeout.

Quickly, I see two ways to fix this.
A- Make `MAGPIE_STARTUP_TIME` dynamic, instead of a user fixed variable. For Moab, we could use the walltime reported by `checkjob` as a startup time in `Magpie_wait_script` function. Something along these lines:

```
walltime=$(checkjob ${MOAB_JOBID} | grep -Po '(?<=WallTime: \s).*' | cut -d' ' -f1))
startuptime=$((10#${walltime:0:2}*60 + 10#${walltime:3:2} + 10#${walltime:6:2}>0))
scriptsleepamounttemp=`expr ${MAGPIE_TIMELIMIT_MINUTES} - ${startuptime}`
```

B- Replace Magpie_wait_script by a mechanism of signal catching. Moab can be told to send a pre-termination signal at the desired time before expiration of the job's wall clock limit, for example:

```
-l signal=SIGHUP@5:00
```

This signal could be catch with the bash trap command and the script killed adequately.

Both solutions are specific to Moab for now, but I think that the mechanisms used are provided by most scheduler.

If I had to choose, I would opt for solution B which I find more elegant as it alleviates the necessity of having a timeout stop watch in Magpie.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the Magpie_wait_script function and the current MAGPIE_STARTUP_TIME handling. Compare the proposed dynamic walltime calculation using Moab's checkjob output with the signal-catching approach, then determine which scheduler behavior the project should support. Done means shutdown timing uses the available walltime reliably without prematurely ending the job.

Written by the indexing model from the issue text.

Assessment

Tech stack
shell
Domain
hpc
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.