futureverse / futureverse/future.batchtools

setting 1 job per worker core after chunking the elements

Ouverte
#39 14 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
R
Étoiles
87
Forks
10
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

I am using an sge template.

Is there a native way in future plan to have elements of a list be allocated to each available core in preset number of workers (ie chunking the list)?

ie if i have a workers with 8 cores and a vector (x) of length 16, i want future to send each element to a core on the workers to run in parallel. (assuming i am sending out jobs from the remote master)

```r
sge <- future::tweak(
future.batchtools::batchtools_sge,
template = 'batchtools.sge-mrg.tmpl',
workers = 2,
resources = list(slots = 1)
)

future::plan(list(sge))

x <- vector('list',16)

ret <- furrr::future_map(x, .f = foo)

```

```bash
#!/bin/bash

## The name of the job, can be anything, simply used when displaying the list of running jobs
#$ -N <%= job.name %>

## Combining output/error messages into one file
#$ -j y

## Giving the name of the output log file
#$ -o <%= log.file %>

## One needs to tell the queue system to use the current directory as the working directory
## Or else the script may fail as it will execute in your top level home directory /home/username
#$ -cwd

## Use environment variables
#$ -V

## Use correct queue
##$ -q all.q

#$ -pe smp <%= resources[["slots"]] %>

## Export value of DEBUGME environemnt var to slave
export DEBUGME=<%= Sys.getenv("DEBUGME") %>

<%= sprintf("export OMP_NUM_THREADS=%i", resources$omp.threads) -%>
<%= sprintf("export OPENBLAS_NUM_THREADS=%i", resources$blas.threads) -%>
<%= sprintf("export MKL_NUM_THREADS=%i", resources$blas.threads) -%>

Rscript -e 'batchtools::doJobCollection("<%= uri %>")'
exit 0

```

cc @wlandau

Guide de contribution

Ouvrir le guide de contribution

Piste de recherche

Start with the future::tweak(... batchtools_sge ...) plan and furrr::future_map example in the issue, then inspect how resources[["slots"]] is passed through batchtools.sge-mrg.tmpl. Done means determining whether native chunking can allocate 16 elements across workers and cores, and recording the supported behavior or requirements.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
r
Domaine
distributed-systems, hpc
Type d'issue
Fonctionnalité
Difficulté
5/5
Temps estimé
Plus d'une semaine
Activité
À l'abandon
Clarté
Plutôt claire
Accessibilité débutants
32/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.