futureverse / futureverse/future.batchtools

setting 1 job per worker core after chunking the elements

Open
#39 14 comments 0 reactions 0 assignees View on GitHub
Dominant language
R
Stars
87
Forks
10
PR merge metrics
No merged PRs in 30d

Description

I am using an sge template.

Is there a native way in future plan to have elements of a list be allocated to each available core in preset number of workers (ie chunking the list)?

ie if i have a workers with 8 cores and a vector (x) of length 16, i want future to send each element to a core on the workers to run in parallel. (assuming i am sending out jobs from the remote master)

```r
sge <- future::tweak(
future.batchtools::batchtools_sge,
template = 'batchtools.sge-mrg.tmpl',
workers = 2,
resources = list(slots = 1)
)

future::plan(list(sge))

x <- vector('list',16)

ret <- furrr::future_map(x, .f = foo)

```

```bash
#!/bin/bash

## The name of the job, can be anything, simply used when displaying the list of running jobs
#$ -N <%= job.name %>

## Combining output/error messages into one file
#$ -j y

## Giving the name of the output log file
#$ -o <%= log.file %>

## One needs to tell the queue system to use the current directory as the working directory
## Or else the script may fail as it will execute in your top level home directory /home/username
#$ -cwd

## Use environment variables
#$ -V

## Use correct queue
##$ -q all.q

#$ -pe smp <%= resources[["slots"]] %>

## Export value of DEBUGME environemnt var to slave
export DEBUGME=<%= Sys.getenv("DEBUGME") %>

<%= sprintf("export OMP_NUM_THREADS=%i", resources$omp.threads) -%>
<%= sprintf("export OPENBLAS_NUM_THREADS=%i", resources$blas.threads) -%>
<%= sprintf("export MKL_NUM_THREADS=%i", resources$blas.threads) -%>

Rscript -e 'batchtools::doJobCollection("<%= uri %>")'
exit 0

```

cc @wlandau

Contributor guide

Open the contributing guide

Research direction

Start with the future::tweak(... batchtools_sge ...) plan and furrr::future_map example in the issue, then inspect how resources[["slots"]] is passed through batchtools.sge-mrg.tmpl. Done means determining whether native chunking can allocate 16 elements across workers and cores, and recording the supported behavior or requirements.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
distributed-systems, hpc
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.