Too many jobs causes "BatchtoolsExpiration: Future ('<none>') expired"
Nobody has claimed this yet.
- Dominant language
- R
- Stars
- 184
- Forks
- 53
- Avg merge
- 7d 2h
- Merged PRs (30d)
- 1
Description
Whenever I run approx. > 200 jobs on our SGE cluster, I get the following error:
BatchtoolsExpiration: Future ('<none>') expired (registry path /ebio/abt3_projects/software/dev/DeepMAsED/notebooks/04_contig_viewing/.future/20190812_101254-P53iu9/batchtools_2061698801)
If I manually batch the jobs into smaller groups (eg., 50 or 100 jobs), then I don't get this error.
Version: r-batchtools 0.9.11 r341hc070d10_0 conda-forge
~/.batchtools.conf.R file:
default.resources = list(h_rt = '00:59:00',
h_vmem = '4G',
threads = '1',
conda.env = 'py3')
cluster.functions = makeClusterFunctionsSGE(template = "~/.batchtools.tmpl")
temp.dir = "/ebio/abt3_projects/temp_data/"
.batchtools.tmpl file:
#!/bin/bash
#$ -N <%= job.name %
#$ -j y
#$ -o <%= log.file %>
#$ -cwd
#$ -V
#$ -pe parallel <%= resources$threads %>
#$ -l h_rt=<%= resources$h_rt %>
#$ -l h_vmem=<%= resources$h_vmem %>
. ~/.bashrc
conda activate <%= resources$conda.env %>
## Export value of DEBUGME environemnt var to slave
export DEBUGME=<%= Sys.getenv("DEBUGME") %>
<%= sprintf("export OMP_NUM_THREADS=%i", resources$omp.threads) -%>
<%= sprintf("export OPENBLAS_NUM_THREADS=%i", resources$blas.threads) -%>
<%= sprintf("export MKL_NUM_THREADS=%i", resources$blas.threads) -%>
Rscript -e 'batchtools::doJobCollection("<%= uri %>")'
exit 0
Example of resources used:
resources = list(h_rt = '00:59:00',
h_vmem = '4G',
threads = 1,
conda.env = 'py3_batchtools') # conda env with batchtools installed
plan(batchtools_sge, resources=resources, workers=50)
I'm using future_lapply() to run a function that calls system2()
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the failure with future_lapply(), system2(), and more than 200 jobs using the shown SGE configuration. Read ~/.batchtools.conf.R and .batchtools.tmpl, then inspect how plan(batchtools_sge, workers=50) submits and tracks job collections. Done means the workload completes without BatchtoolsExpiration while preserving the reported resource settings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- hpc
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100