futureverse / futureverse/parallelly
behavior of `availableWorkers` when Slurm `--nodes`, `--ntasks` and `--cpus-per-task` are provided
- Dominant language
- R
- Stars
- 140
- Forks
- 9
- PR merge metrics
- No merged PRs in 30d
Description
I noticed that in this case, available workers is based on `-c` (`--cpus-per-task`) and not `-n` (`--ntasks`).
```bash
paciorek@smeagol:~> srun --ntasks=4 --cpus-per-task=2 --pty bash
paciorek@scf-sm11:~> Rscript -e "parallelly::availableWorkers()"
[1] "scf-sm11" "scf-sm11"
```
I think in this case, it would be more natural to have it report 4 workers, with the user probably wanting to use threading (e.g., linear algebra or Rcpp+openMP) within each of the 4 workers. That said, there may be cases where returning 2 available workers makes most sense.
However, the result above seems inconsistent with this next result (after all, why should it matter how many nodes the 4 tasks are running on?):
```bash
paciorek@smeagol:~> srun --nodes=2 --ntasks=4 --cpus-per-task=2 --pty bash
paciorek@scf-sm10:~> Rscript -e "parallelly::availableWorkers()"
[1] "scf-sm10" "scf-sm10" "scf-sm11" "scf-sm11"
```
That said, I can imagine handling all the various ways a user might use `--nodes` (`-N`), `--ntasks` (`-n`), and `--cpus-per-task` (`-c`) might be tricky...
EDIT 2022-12-13: Add long options formats for clarification. /Henrik
Contributor guide
Research direction
Start at parallelly::availableWorkers() and reproduce the two srun examples with --nodes, --ntasks, and --cpus-per-task. Trace how Slurm allocation values are interpreted, then make the worker result consistent with the intended task-versus-CPU semantics across one and multiple nodes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100