futureverse / futureverse/parallelly

HELP WANTED: Agility of availableCores()

Open
#17 15 comments 0 reactions 0 assignees View on GitHub
enhancement feature/resources help wanted
Dominant language
R
Stars
140
Forks
9
PR merge metrics
No merged PRs in 30d

Description

`parallelly::availableCores()` returns the number of cores available for multicore processing. R itself provides `parallel::detectCores()` for this. There is also the `mc.cores` option (default to environment variable `MC_CORES`) set when `parallel` is loaded. Beyond this, various systems/setups set specific environment variables to reflect the number of available/allocated cores. For instance, resource manager PBS, sets environment variable `PBS_NUM_PPN` on the compute node specifying the number of allotted cores.

Currently, `availableCores()` defaults to return the first valid value of (in order):
1. `PBS_NUM_PPN`
2. `mc.cores` (and `MC_CORES`)
3. `parallel::detectCores()`

I would like to add support for more resource/workload managers and other distributed processing environments. For instance,
- [x] [PBS](https://en.wikipedia.org/wiki/Portable_Batch_System) (Portable Batch System)
- [x] `PBS_NUM_NODES`,
- [x] `PBS_NUM_PPN`,
- [x] ~~`PBS_NP`~~ (see comment below)
- [x] [PBSPro](https://github.com/PBSPro/pbspro)
- [x] `NCPUS`
- [x] [Slurm](https://en.wikipedia.org/wiki/Slurm_Workload_Manager) (Simple Linux Utility for Resource Management)
- Manual: https://slurm.schedmd.com/
- [x] `SLURM_CPUS_PER_TASK`, e.g. `--cpus-per-task=3` (or short `-N 3`)
- [x] `SLURM_CPUS_ON_NODE`, e.g. `--ntasks=3` (or short `-n 3`): only truthworty when `--nodes=1`
- [ ] Support for multi-node job requests, e.g. `--nodes=2 --ntasks=3`. Identify the number of cores to run on the current node.
- [x] [Oracle Grid Engine](https://en.wikipedia.org/wiki/Oracle_Grid_Engine) (aka Sun Grid Engine, CODINE, GRD). Environment variable: `NSLOTS` (?), cf. http://cc.in2p3.fr/docenligne/969
- [ ] LSF/[OpenLava](https://en.wikipedia.org/wiki/OpenLava) (Platform Load Sharing Facility).
- [x] `LSB_DJOB_NUMPROC` - "The number of processors (slots) allocated to the job." (), cf. PR HenrikBengtsson/future#360
- [ ] `LSB_MAX_NUM_PROCESSORS`(?) - "The maximum number of processors requested when the job is submitted. [...]
Set during job execution based on bsub options. For example, for a job submitted with -n 2,4, the maximum number of processors requested is 4." (). This setting is recommended by https://grid.rcs.hbs.org/parallel-r
- [ ] ...?
- [ ] [Spark](https://spark.apache.org/)
- [ ] [OAR](http://oar.imag.fr)
- [ ] [HTCondor](https://en.wikipedia.org/wiki/HTCondor)
- Handled in Issue #50
- [ ] [Moab](https://en.wikipedia.org/wiki/Moab_Cluster_Suite)
- [x] PJM (https://staff.cs.manchester.ac.uk/~fumie/internal/Job_Operation_Software_en.pdf)
- [x] `PJM_VNODE_CORE`
- [x] `PJM_PROC_BY_NODE` (not always)

What else am I missing?

- [x] [BBS_HOME](https://github.com/Bioconductor/BiocParallel/blob/9019d88437d9166ec8faab27be4e2063bdde6a93/R/SnowParam-class.R#L67-L68) - (env var) if set (to anything) BiocParallel will "throttle number of cores used on Bioconductor build systems [...] to 4".
- [ ] [snowFT](https://cran.r-project.org/web/packages/snowFT)'s `./.clustersize` file that specifies number of workers?

I appreciate any feedback on what environment variables or commands that are available to a compute node to query the number of allotted cores, iff at all. Please try to provide links to documentations if you can.
# References
- http://slurm.schedmd.com/rosetta.pdf

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.