futureverse / futureverse/future.batchtools

Problem forwarding batchtools resources to individual futures

Open
#75 4 comments 0 reactions 0 assignees View on GitHub
feature/resources
Dominant language
R
Stars
87
Forks
10
PR merge metrics
No merged PRs in 30d

Description

## Summary

When I supply a `resources` list to a `batchtools` future, the resources seem to be ignored. I was alerted to this via `targets`, https://github.com/ropensci/targets/discussions/562#discussioncomment-1304379 and https://github.com/ropensci/targets/discussions/632 (cc @wresch, @sdechaumet)

## Reproducible example

Using an SGE cluster with this template:

```
#!/bin/bash
#$ -cwd
#$ -j y
#$ -o <%= log.file %>
#$ -V
#$ -N <%= job.name %>
#$ -pe smp <%= resources$slots %>
module load R/4.0.3
Rscript -e 'batchtools::doJobCollection("<%= uri %>")'
exit 0
```

R console:

```r
library(future)
library(future.batchtools)
plan(batchtools_sge, template = "sge_batchtools.tmpl")
f <- future(Sys.sleep(5), resources = list(slots = 2))
# Error: Fatal error occurred: 101. Command 'qsub' produced exit code 2. Output: 'Unable to read script file because of error: ERROR! -pe option must have range as 2nd argument'
```

When I run `debug(BatchtoolsFuture)` on this example, I see that `resources = list(slots = 2)` is not actually passed to the `BatchtoolsFuture()` function. `BatchtoolsFuture()` appears to receive the default `resources = list()`.

## Expected behavior

An SGE job with 2 slots should be submitted.

## Session information

I am using the following commits of `future` and `future.batchtools`:

* https://github.com/HenrikBengtsson/future/commit/344a97f1cc85283095a43fe01cfabf3fb33fdf70
* https://github.com/HenrikBengtsson/future.batchtools/commit/ea3b86cbc6b30f62f78cfd9ef7c5a64f5d42e683

```r
> sessionInfo()
R version 4.0.3 (2020-10-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Red Hat Enterprise Linux

Matrix products: default
BLAS/LAPACK:

locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] future.batchtools_0.10.0-9000 future_1.22.1-9000

loaded via a namespace (and not attached):
[1] parallelly_1.28.1 magrittr_2.0.1 hms_1.1.0 progress_1.2.2
[5] rappdirs_0.3.3 debugme_1.1.0 R6_2.5.1 brew_1.0-6
[9] rlang_0.4.11 fansi_0.5.0 globals_0.14.0 tools_4.0.3
[13] parallel_4.0.3 checkmate_2.0.0 data.table_1.14.0 utf8_1.2.2
[17] withr_2.4.2 ellipsis_0.3.2 base64url_1.4 digest_0.6.27
[21] tibble_3.1.4 lifecycle_1.0.0 crayon_1.4.1 fs_1.5.0
[25] vctrs_0.3.8 batchtools_0.9.15 codetools_0.2-18 stringi_1.7.4
[29] pillar_1.6.2 compiler_4.0.3 backports_1.2.1 prettyunits_1.1.1
[33] listenv_0.8.0 pkgconfig_2.0.3
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.