dselivanov / dselivanov/text2vec
WISH: Less aggressive parallelization by default (please don't use *all* CPU cores)
- Dominant language
- R
- Stars
- 876
- Forks
- 133
- PR merge metrics
- No merged PRs in 30d
Description
Hi, I noticed **text2vec** runs on _all_ CPU cores by default on Unix. This is from:
https://github.com/dselivanov/text2vec/blob/9ddf836b995511d8747cc98f753e9cc706cf3c84/R/zzz.R#L6-L9
https://github.com/dselivanov/text2vec/blob/9ddf836b995511d8747cc98f753e9cc706cf3c84/R/mc_queue.R#L1-L4
Defaulting to all cores causes major problems on machines used by multiple users, but also when there are software tools running at the same time. I spotted this on a 128 CPU core machine. Imagine running another 10-20 processes like that at the same time on this machine - it'll quickly come to a halt, which is a real problem.
Although the behavior can be changed by setting an R option, many users are not aware of the problem ... until the sysadms yell at them. Also, **text2vec** might be running deep down as a dependency that other package maintainers might not be aware of, so this behavior might be inherited also be other packages without them knowing.
Could you please consider switch the default to be more conservatively. Personally, I'm in the camp that everything should run sequentially (single-core), unless the user configures it otherwise. CRAN has a limit of two CPU cores.
(Disclaimer: I'm the author) If you don't want to do this, could you please consider changing from:
```r
parallel::detectCores(logical = FALSE)
```
to
```r
parallelly::availableCores(logical = FALSE)
```
because the latter gives sysadms a chance to limit it on their end, and it also respects CGroups settings, job scheduler allocations, etc. Please see https://parallelly.futureverse.org/#availablecores-vs-paralleldetectcores for more details.
Thank you
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.