futureverse / futureverse/future.batchtools
setting 1 job per worker core after chunking the elements
- Langage dominant
- R
- Étoiles
- 87
- Forks
- 10
- Métriques de merge des PR
- Aucune PR mergée en 30 j
Description
I am using an sge template.
Is there a native way in future plan to have elements of a list be allocated to each available core in preset number of workers (ie chunking the list)?
ie if i have a workers with 8 cores and a vector (x) of length 16, i want future to send each element to a core on the workers to run in parallel. (assuming i am sending out jobs from the remote master)
```r
sge <- future::tweak(
future.batchtools::batchtools_sge,
template = 'batchtools.sge-mrg.tmpl',
workers = 2,
resources = list(slots = 1)
)
future::plan(list(sge))
x <- vector('list',16)
ret <- furrr::future_map(x, .f = foo)
```
```bash
#!/bin/bash
## The name of the job, can be anything, simply used when displaying the list of running jobs
#$ -N <%= job.name %>
## Combining output/error messages into one file
#$ -j y
## Giving the name of the output log file
#$ -o <%= log.file %>
## One needs to tell the queue system to use the current directory as the working directory
## Or else the script may fail as it will execute in your top level home directory /home/username
#$ -cwd
## Use environment variables
#$ -V
## Use correct queue
##$ -q all.q
#$ -pe smp <%= resources[["slots"]] %>
## Export value of DEBUGME environemnt var to slave
export DEBUGME=<%= Sys.getenv("DEBUGME") %>
<%= sprintf("export OMP_NUM_THREADS=%i", resources$omp.threads) -%>
<%= sprintf("export OPENBLAS_NUM_THREADS=%i", resources$blas.threads) -%>
<%= sprintf("export MKL_NUM_THREADS=%i", resources$blas.threads) -%>
Rscript -e 'batchtools::doJobCollection("<%= uri %>")'
exit 0
```
cc @wlandau
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Start with the future::tweak(... batchtools_sge ...) plan and furrr::future_map example in the issue, then inspect how resources[["slots"]] is passed through batchtools.sge-mrg.tmpl. Done means determining whether native chunking can allocate 16 elements across workers and cores, and recording the supported behavior or requirements.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- r
- Domaine
- distributed-systems, hpc
- Type d'issue
- Fonctionnalité
- Difficulté
- 5/5
- Temps estimé
- Plus d'une semaine
- Activité
- À l'abandon
- Clarté
- Plutôt claire
- Accessibilité débutants
- 32/100