mlr-org / mlr-org/batchtools

Old version of Slurm, "invalid job state specified: SI"

Open
#300 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
R
Stars
184
Forks
53
Avg merge
7d 2h
Merged PRs (30d)
1

Description

Wanted to report this issue that I have "solved." Was using batchtools via future.batchtools but got this cryptic (to me) error when I changed employers and thus moved from a PBS system to a Slurm system.

Error: Failed to submit BatchtoolsSlurmFuture (<none>). The reason was: Listing of jobs failed (exit code 1);
cmd: 'squeue --user=$USER --states=R,S,CG,RS,SI,SO,ST --noheader --format=%i -r'
output:
squeue: error: Invalid job state specified: SI
squeue: error: Valid job states include: PENDING,RUNNING,SUSPENDED,COMPLETED,CANCELLED,FAILED,TIMEOUT,NODE_FAIL,PREEMPTED,BOOT_FAIL,DEADLINE,COMPLETING,CONFIGURING,RESIZING,SPECIAL_EXIT

I initially thought the error message was spurious since I can see online that SI is a valid job state. After a lot of poking and prodding, it occurred to me that maybe different versions of Slurm accept different job states arguments. Indeed this was the issue. My employer is running Slurm 16.0.5, which is nearly 7 years old and in that time period, the SI state was added.

Maybe I don't understand batchtools functionality well enough, but it seemed I couldn't override the squeue --user=$USER --states=R,S,CG,RS,SI,SO,ST --noheader --format=%i -r called by makeClusterFunctionsSlurm() (https://github.com/mllg/batchtools/blob/1196047ed5115d54bde2923848c1f3ec11fda6d2/R/clusterFunctionsSlurm.R#L106).

My solution was to install the most recent version of batchtools that did not use "SI" in that argument as well as an older version of future.batchtools that didn't force an update to an incompatible version of batchtools.

I suspect a pretty strong chance that there's a better way for the end user to deal with this than what I came up with, but thought the package dev(s) should be aware that this is a possible issue. Worst case, the next person searching the internet for help with this problem might find this :)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with R/clusterFunctionsSlurm.R around the makeClusterFunctionsSlurm() command at line 106 and inspect how the squeue state list is selected. Check compatibility with Slurm 16.0.5 and determine whether an existing configuration path can avoid the unsupported SI state. Done should provide a workable path for older Slurm installations or clearly document the required version and workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
hpc
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.