Enhancement proposal: making timeout behavior more dynamic
Nobody has claimed this yet.
- Dominant language
- Shell
- Stars
- 198
- Forks
- 51
- PR merge metrics
- No merged PRs in 30d
Description
While testing, I found that when the setup time was shorter than expected, the job was unable to use all the walltime available because of the shutdown timeout.
Quickly, I see two ways to fix this.
A- Make `MAGPIE_STARTUP_TIME` dynamic, instead of a user fixed variable. For Moab, we could use the walltime reported by `checkjob` as a startup time in `Magpie_wait_script` function. Something along these lines:
```
walltime=$(checkjob ${MOAB_JOBID} | grep -Po '(?<=WallTime: \s).*' | cut -d' ' -f1))
startuptime=$((10#${walltime:0:2}*60 + 10#${walltime:3:2} + 10#${walltime:6:2}>0))
scriptsleepamounttemp=`expr ${MAGPIE_TIMELIMIT_MINUTES} - ${startuptime}`
```
B- Replace Magpie_wait_script by a mechanism of signal catching. Moab can be told to send a pre-termination signal at the desired time before expiration of the job's wall clock limit, for example:
```
-l signal=SIGHUP@5:00
```
This signal could be catch with the bash trap command and the script killed adequately.
Both solutions are specific to Moab for now, but I think that the mechanisms used are provided by most scheduler.
If I had to choose, I would opt for solution B which I find more elegant as it alleviates the necessity of having a timeout stop watch in Magpie.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the Magpie_wait_script function and the current MAGPIE_STARTUP_TIME handling. Compare the proposed dynamic walltime calculation using Moab's checkjob output with the signal-catching approach, then determine which scheduler behavior the project should support. Done means shutdown timing uses the available walltime reliably without prematurely ending the job.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- shell
- Domain
- hpc
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100