futureverse / futureverse/parallelly
Linux: detectCores(logical = FALSE) is always equal to detectCores(logical = TRUE)
- Dominant language
- R
- Stars
- 140
- Forks
- 9
- PR merge metrics
- No merged PRs in 30d
Description
# Issue
On Linux systems(*), `parallel::detectCores()` ignores the `logical` argument. For example, on my notebook with four physical CPU cores with two "processors" each, I get:
```r
> parallel::detectCores(logical = FALSE)
[1] 8
> parallel::detectCores(logical = TRUE) ## default
[1] 8
```
I think the former should really be `4`.
FWIW, `help("detectCores", package = "parallel")` acknowledges this:
> `logical` | Logical: if possible, use the number of physical CPUs/cores (if FALSE) or logical CPUs (if TRUE). Currently this is honoured only on macOS, Solaris and Windows.
(*) A Linux system is where `R.version$os` starts with `linux`, e.g. in R 4.2.2 on my Ubuntu 20.04 system I have `R.version$os == "linux-gnu"`.
# Troubleshooting
In both cases, [`detectCores()` uses the following `system(..., intern = TRUE)` call on Linux](https://github.com/wch/r-source/blob/fd9bdecb37287dc6078b1b6f1f43fc8a52c2bc78/src/library/parallel/R/detectCores.R#L35):
```sh
$ grep "^processor" /proc/cpuinfo 2>/dev/null | wc -l
8
```
to get the number of cores - logical or not.
If I look at `lscpu` (not installed on all systems), I get:
```sh
$ lscpu
…
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
…
```
on my machine. That suggests there are 8 logical CPU cores, and 4*1 = 4 physical ones. We can query the raw data for this from `/proc/cpuinfo` as:
```sh
$ cat /proc/cpuinfo | grep -E "^(processor|core id)"
processor : 0
core id : 0
processor : 1
core id : 1
processor : 2
core id : 2
processor : 3
core id : 3
processor : 4
core id : 0
processor : 5
core id : 1
processor : 6
core id : 2
processor : 7
core id : 3
```
which explains:
```sh
$ grep "^processor" /proc/cpuinfo 2>/dev/null | wc -l
8
```
for counting the number of logical CPU cores. However, for the physical ones, I think we should count the unique number of core IDs;
```sh
$ grep "^core id" /proc/cpuinfo 2>/dev/null | sort -u | wc -l
4
```
# Action
So, this begs the question, why doesn't `parallel::detectCores()` do this? I'm pretty sure this has been discussed somewhere before. The first task is to identify any discussions and rationales for the current implementation.
# See also
* This is related to Issue #47 (`detectThreadsPerCore()`)
* It's complicated, e.g. https://superuser.com/questions/378273/strange-cpuinfo-how-many-cores-does-these-physical-cpu-has/932418#932418
* R-devel thread 'Detecting physical CPUs in detectCores() on Linux platforms' started on 2023-08-07 ().
Contributor guide
Research direction
Begin with parallel::detectCores() and the Linux /proc/cpuinfo commands documented in the issue; review Issue #47 and the linked R-devel thread for prior rationale. Done means documenting the existing constraints and reaching a clearly supported approach for distinguishing logical from physical CPUs on Linux.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100