mlr-org / mlr-org/batchtools

makeRegistry(NA) causes jobs to expire on Slurm cluster

Open
#237 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
R
Stars
184
Forks
53
Avg merge
7d 2h
Merged PRs (30d)
1

Description

Hi,

I've just been working on setting up batchtools on a Slurm cluster. I was finding that all my jobs were listed as 'Expired' after running getStatus() and it took a while to figure out why. Seems to be because I was following the 'Getting Started' vignette, which suggests running:

makeRegistry(NA)

But that creates the registry in a temporary folder. So far as I understand, the temporary folder gets created on the login node, and is not accessible from the worker nodes. So once the worker nodes are called, they cannot access the job files, and so expire (without being able to write a log file explaining what went wrong).

I don't have access to any other Slurm clusters, so don't know if this is the default behaviour with Slurm clusters or something specific to mine. To save other people running through this, could I suggest either (A) changing the 'Getting started' guide to have makeRegistry() use a random folder in the home directory, or (B) change the behaviour of makeRegistry on Slurm clusters to avoid temporary folders?

Thanks for developing this great package!

Nathan

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Getting Started vignette and the makeRegistry(NA) and getStatus() workflow described in the issue. Reproduce the registry location behavior on a Slurm cluster, then determine whether the documented example or makeRegistry behavior is the appropriate scope; done means jobs no longer expire because workers cannot access the registry files.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
hpc
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.