futureverse / futureverse/future.batchtools

setting 1 job per worker core after chunking the elements

Abierto
#39 14 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
R
Estrellas
87
Forks
10
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

I am using an sge template.

Is there a native way in future plan to have elements of a list be allocated to each available core in preset number of workers (ie chunking the list)?

ie if i have a workers with 8 cores and a vector (x) of length 16, i want future to send each element to a core on the workers to run in parallel. (assuming i am sending out jobs from the remote master)

```r
sge <- future::tweak(
future.batchtools::batchtools_sge,
template = 'batchtools.sge-mrg.tmpl',
workers = 2,
resources = list(slots = 1)
)

future::plan(list(sge))

x <- vector('list',16)

ret <- furrr::future_map(x, .f = foo)

```

```bash
#!/bin/bash

## The name of the job, can be anything, simply used when displaying the list of running jobs
#$ -N <%= job.name %>

## Combining output/error messages into one file
#$ -j y

## Giving the name of the output log file
#$ -o <%= log.file %>

## One needs to tell the queue system to use the current directory as the working directory
## Or else the script may fail as it will execute in your top level home directory /home/username
#$ -cwd

## Use environment variables
#$ -V

## Use correct queue
##$ -q all.q

#$ -pe smp <%= resources[["slots"]] %>

## Export value of DEBUGME environemnt var to slave
export DEBUGME=<%= Sys.getenv("DEBUGME") %>

<%= sprintf("export OMP_NUM_THREADS=%i", resources$omp.threads) -%>
<%= sprintf("export OPENBLAS_NUM_THREADS=%i", resources$blas.threads) -%>
<%= sprintf("export MKL_NUM_THREADS=%i", resources$blas.threads) -%>

Rscript -e 'batchtools::doJobCollection("<%= uri %>")'
exit 0

```

cc @wlandau

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.