mlr-org / mlr-org/batchtools

Too many jobs causes "BatchtoolsExpiration: Future ('<none>') expired"

Open
#240 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
R
Stars
184
Forks
53
Avg merge
7d 2h
Merged PRs (30d)
1

Description

Whenever I run approx. > 200 jobs on our SGE cluster, I get the following error:

BatchtoolsExpiration: Future ('<none>') expired (registry path /ebio/abt3_projects/software/dev/DeepMAsED/notebooks/04_contig_viewing/.future/20190812_101254-P53iu9/batchtools_2061698801)

If I manually batch the jobs into smaller groups (eg., 50 or 100 jobs), then I don't get this error.

Version: r-batchtools 0.9.11 r341hc070d10_0 conda-forge

~/.batchtools.conf.R file:

default.resources = list(h_rt = '00:59:00',
                         h_vmem = '4G',
			 threads = '1',
                         conda.env = 'py3')
cluster.functions = makeClusterFunctionsSGE(template = "~/.batchtools.tmpl")
temp.dir = "/ebio/abt3_projects/temp_data/"

.batchtools.tmpl file:

#!/bin/bash
#$ -N <%= job.name %
#$ -j y
#$ -o <%= log.file %>
#$ -cwd
#$ -V
#$ -pe parallel <%= resources$threads %>
#$ -l h_rt=<%= resources$h_rt %>
#$ -l h_vmem=<%= resources$h_vmem %>


. ~/.bashrc
conda activate <%= resources$conda.env %>

## Export value of DEBUGME environemnt var to slave
export DEBUGME=<%= Sys.getenv("DEBUGME") %>

<%= sprintf("export OMP_NUM_THREADS=%i", resources$omp.threads) -%>
<%= sprintf("export OPENBLAS_NUM_THREADS=%i", resources$blas.threads) -%>
<%= sprintf("export MKL_NUM_THREADS=%i", resources$blas.threads) -%>

Rscript -e 'batchtools::doJobCollection("<%= uri %>")'
exit 0

Example of resources used:

resources = list(h_rt = '00:59:00',
                 h_vmem = '4G',
                 threads = 1,
                 conda.env = 'py3_batchtools')     # conda env with batchtools installed
plan(batchtools_sge, resources=resources, workers=50)

I'm using future_lapply() to run a function that calls system2()

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure with future_lapply(), system2(), and more than 200 jobs using the shown SGE configuration. Read ~/.batchtools.conf.R and .batchtools.tmpl, then inspect how plan(batchtools_sge, workers=50) submits and tracks job collections. Done means the workload completes without BatchtoolsExpiration while preserving the reported resource settings.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
hpc
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.