slurm: Socket timed out on send/recv operation

Hello! I recently posted an issue on the future.batchtools repo and was advised that it might be better addressed here. The full contents are included below:

Background and details:

I am currently developing a package which includes a function that uses future_lapply. I have a script that performs some testing (i.e. loads packages and data, plans the future, and calls the function in my package that utilises future_lapply). I run the script on the log-in node of the remote HPC as follows:

Rscript --vanilla my_script.R &> logs.txt &

Within my_script.R the future is planned with the following:

future::plan(
  future.batchtools::batchtools_slurm,
  template = "/crex/proj/snic2018-8-151/private/batchtools.slurm.tmpl",
  resources = list(
    account = "snic2018-8-151", partition = "core", ntasks = 1L,
    time = "24:00:00", jobname = "testingPoissonSorted",
    modules = "R_packages/3.5.0", R = "R/3.5.0", log.file = file.path(currPath, "logs/slurm.txt")
 ))

And my batchtools template includes:

\#!/bin/bash                                                                                                                                                                                                                                                    

\#SBATCH --job-name=<%= resources$jobname %>                                                                                                                                                                                                                    
\#SBATCH --time=<%= resources$time %>                                                                                                                                                                                                                           
\#SBATCH --ntasks=<%= resources$ntasks %>                                                                                                                                                                                                                       
\#SBATCH --account=<%= resources$account %>                                                                                                                                                                                                                     
\#SBATCH --partition=<%= resources$partition %>                                                                                                                                                                                                                 
\#SBATCH --output=<%= resources$log.file %>                                                                                                                                                                                                                     
                                                                                                                                                                                                                     
source /etc/profile                                                                                                                                                                                  
module add <%= paste(resources$modules, resources$R) %>                                                                                                                                        
Rscript -e 'batchtools::doJobCollection("<%= uri %>")'

Error and "expected" response:

While testing the package on a Slurm-based system I get the following error (reported in logs.txt):

Error in OSError("Listing of jobs failed", res) :
Listing of jobs failed (exit code 1);
cmd: 'squeue --user=$USER --states=R,S,CG --noheader --format=%i -r'
output:
slurm_load_jobs error: Socket timed out on send/recv operation
Calls: spSwarm ... unique -> -> listJobs -> OSError -> stopf
Execution halted

It seems that this arises when the schedular is "busy" and, I am guessing, the communication between batchtools and slurm times out. Since the jobs have already started, they continue to run until completion although, because of the apparent "disconnection" between batchtools and slurm, the output from the run (saved downstream in the script after the call to future_lapply) is never generated. This is somewhat painful for those of us working on a limited number of core hours per month as the hours get "spent" without any "result" being generated.

After a bit of looking around it would seem that the sysadmins may be able to help prevent this (bug report) but, in my humble opinion, it would also be ideal if a) the user can do something to increase the time that future/batchtools waits for a response from the schedular or b) that the jobs are automatically canceled if batchtools looses contact with the schedular (potentially problematic since contact is lost).

For the sake of completeness the output from batchtools is included below:

WARNING: ignoring environment value of R_HOME
[bt]: This is batchtools v0.9.10
[bt]: Starting calculation of 1 jobs
[bt]: Setting working directory to '/crex/proj/snic2018-8-151/private/CIMseq/testingPoissonMouse'
Loading required package: sp.scRNAseq
[bt]: Memory measurement disabled
[bt]: Starting job [batchtools job.id=1]
[bt]: Setting seed to 1 ...

[bt]: Job terminated successfully [batchtools job.id=1]
[bt]: Calculation finished!

[bt]: Job terminated successfully [batchtools job.id=1]
[bt]: Calculation finished!

Disclaimer:

I wouldn't consider myself an experienced HPC/future/batchtools user and this could be potentially "all my fault" and painfully obvious to more experienced users and, if that is the case, hopefully submitting this issue will guide others of a similar experience level in the right direction in the future .

Kind Regards,
Jason

Start with the reported squeue --user=$USER --states=R,S,CG --noheader --format=%i -r failure and the listJobs call shown in the batchtools stack trace. Trace how future_lapply and batchtools handle a Slurm communication timeout, then determine whether the intended behavior is configurable waiting or cancellation. The issue does not name a source file or test, and completion criteria need agreement.