Mutlicore does not work properly (Warning in selectChildren)
Nobody has claimed this yet.
- Dominant language
- R
- Stars
- 184
- Forks
- 53
- Avg merge
- 7d 2h
- Merged PRs (30d)
- 1
Description
I am having some trouble with the Multicore cluster for parallel computation. It seems that once there are more jobs than CPU's submitJobs() keeps running but does not submit more jobs (i.e. no CPU usage). Here a mini example.
library("batchtools") # Version 0.9.11
tmp = makeRegistry(file.dir = NA, make.default = FALSE)
tmp$cluster.functions = makeClusterFunctionsMulticore(1)
piApprox = function(n) {
nums = matrix(runif(2 * n), ncol = 2)
d = sqrt(nums[, 1]^2 + nums[, 2]^2)
4 * mean(d <= 1)
}
batchMap(fun = piApprox, n = rep(1e5, 2), reg = tmp)
# submitJobs(ids = 1, reg = tmp) # Works
submitJobs(ids = findJobs(reg = tmp), reg = tmp) # Gets stuck
Once I stop the execution, there is multiple warnings like the ones described in #221 (see below). However, in my case jobs will not be completed which makes it (at least for me) quite annoying.
Submitting 2 jobs in 2 chunks using cluster functions 'Multicore' ...
Warnmeldungen:
1: In selectChildren(jobs, timeout) :
cannot wait for child 22536 as it does not exist
2: In selectChildren(jobs, timeout) :
cannot wait for child 22536 as it does not exist
I already found a fix, but I am not sure if it is a good one. I changed line 63 of 'clusterFunctionsMulticore.R' as follows.
self$jobs = self$jobs[count <= 1L] # old
self$jobs = self$jobs[count < 1L] # new
With this change it seems to work again. I got the impression that in the original version it does not reduce the self$jobs table because the value for count seems to be originally 0 and then count + 1 is also <= 1. Thus, it gets stuck in the repeat loop. However, with a quick search I didn't find anything in your commits that explains this (although it worked back in the day for me.). Here is some additional info on my setup.
> sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Manjaro Linux
Matrix products: default
BLAS: /usr/lib/libopenblasp-r0.3.6.so
LAPACK: /usr/lib/liblapack.so.3.8.0
locale:
[1] LC_CTYPE=de_DE.UTF-8 LC_NUMERIC=C LC_TIME=de_DE.UTF-8 LC_COLLATE=de_DE.UTF-8
[5] LC_MONETARY=de_DE.UTF-8 LC_MESSAGES=de_DE.UTF-8 LC_PAPER=de_DE.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=de_DE.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] batchtools_0.9.11 data.table_1.12.2
loaded via a namespace (and not attached):
[1] Rcpp_1.0.1 prettyunits_1.0.2 withr_2.1.2 assertthat_0.2.1 digest_0.6.18 crayon_1.3.4
[7] rappdirs_0.3.1 R6_2.4.0 backports_1.1.4 magrittr_1.5 rlang_0.3.4 progress_1.2.2
[13] stringi_1.4.3 fs_1.3.1 rstudioapi_0.10 brew_1.0-6 checkmate_1.9.1 tools_3.6.1
[19] hms_0.4.2 parallel_3.6.1 compiler_3.6.1 pkgconfig_2.0.2 base64url_1.4
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the hang with the R example using makeClusterFunctionsMulticore(1), two jobs, and submitJobs(ids = findJobs(reg = tmp)). Then inspect clusterFunctionsMulticore.R around line 63 and compare the selectChildren warnings with issue #221. Done means jobs exceeding the CPU count complete without repeated warnings or a stuck submitJobs call.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- hpc
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 42/100