futureverse / futureverse/future.batchtools
setting 1 job per worker core after chunking the elements
- Ngôn ngữ chính
- R
- Star
- 87
- Fork
- 10
- Chỉ số merge pull request
- Không có pull request nào được merge trong 30 ngày
Mô tả
I am using an sge template.
Is there a native way in future plan to have elements of a list be allocated to each available core in preset number of workers (ie chunking the list)?
ie if i have a workers with 8 cores and a vector (x) of length 16, i want future to send each element to a core on the workers to run in parallel. (assuming i am sending out jobs from the remote master)
```r
sge <- future::tweak(
future.batchtools::batchtools_sge,
template = 'batchtools.sge-mrg.tmpl',
workers = 2,
resources = list(slots = 1)
)
future::plan(list(sge))
x <- vector('list',16)
ret <- furrr::future_map(x, .f = foo)
```
```bash
#!/bin/bash
## The name of the job, can be anything, simply used when displaying the list of running jobs
#$ -N <%= job.name %>
## Combining output/error messages into one file
#$ -j y
## Giving the name of the output log file
#$ -o <%= log.file %>
## One needs to tell the queue system to use the current directory as the working directory
## Or else the script may fail as it will execute in your top level home directory /home/username
#$ -cwd
## Use environment variables
#$ -V
## Use correct queue
##$ -q all.q
#$ -pe smp <%= resources[["slots"]] %>
## Export value of DEBUGME environemnt var to slave
export DEBUGME=<%= Sys.getenv("DEBUGME") %>
<%= sprintf("export OMP_NUM_THREADS=%i", resources$omp.threads) -%>
<%= sprintf("export OPENBLAS_NUM_THREADS=%i", resources$blas.threads) -%>
<%= sprintf("export MKL_NUM_THREADS=%i", resources$blas.threads) -%>
Rscript -e 'batchtools::doJobCollection("<%= uri %>")'
exit 0
```
cc @wlandau
Hướng dẫn đóng góp
Hướng nghiên cứu
Start with the future::tweak(... batchtools_sge ...) plan and furrr::future_map example in the issue, then inspect how resources[["slots"]] is passed through batchtools.sge-mrg.tmpl. Done means determining whether native chunking can allocate 16 elements across workers and cores, and recording the supported behavior or requirements.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- r
- Lĩnh vực
- distributed-systems, hpc
- Loại issue
- Tính năng
- Độ khó
- 5/5
- Thời gian dự kiến
- Hơn một tuần
- Mức độ hoạt động
- Đình trệ
- Độ rõ ràng
- Khá rõ ràng
- Mức phù hợp với người mới
- 32/100