futureverse / futureverse/parallelly

Linux: detectCores(logical = FALSE) is always equal to detectCores(logical = TRUE)

Open
#92 0 comments 0 reactions 0 assignees View on GitHub
help wanted question
Dominant language
R
Stars
140
Forks
9
PR merge metrics
No merged PRs in 30d

Description

# Issue

On Linux systems(*), `parallel::detectCores()` ignores the `logical` argument. For example, on my notebook with four physical CPU cores with two "processors" each, I get:

```r
> parallel::detectCores(logical = FALSE)
[1] 8
> parallel::detectCores(logical = TRUE) ## default
[1] 8
```

I think the former should really be `4`.

FWIW, `help("detectCores", package = "parallel")` acknowledges this:

> `logical` | Logical: if possible, use the number of physical CPUs/cores (if FALSE) or logical CPUs (if TRUE). Currently this is honoured only on macOS, Solaris and Windows.

(*) A Linux system is where `R.version$os` starts with `linux`, e.g. in R 4.2.2 on my Ubuntu 20.04 system I have `R.version$os == "linux-gnu"`.

# Troubleshooting

In both cases, [`detectCores()` uses the following `system(..., intern = TRUE)` call on Linux](https://github.com/wch/r-source/blob/fd9bdecb37287dc6078b1b6f1f43fc8a52c2bc78/src/library/parallel/R/detectCores.R#L35):

```sh
$ grep "^processor" /proc/cpuinfo 2>/dev/null | wc -l
8
```

to get the number of cores - logical or not.

If I look at `lscpu` (not installed on all systems), I get:

```sh
$ lscpu

CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1

```

on my machine. That suggests there are 8 logical CPU cores, and 4*1 = 4 physical ones. We can query the raw data for this from `/proc/cpuinfo` as:

```sh
$ cat /proc/cpuinfo | grep -E "^(processor|core id)"
processor : 0
core id : 0
processor : 1
core id : 1
processor : 2
core id : 2
processor : 3
core id : 3
processor : 4
core id : 0
processor : 5
core id : 1
processor : 6
core id : 2
processor : 7
core id : 3
```

which explains:

```sh
$ grep "^processor" /proc/cpuinfo 2>/dev/null | wc -l
8
```

for counting the number of logical CPU cores. However, for the physical ones, I think we should count the unique number of core IDs;

```sh
$ grep "^core id" /proc/cpuinfo 2>/dev/null | sort -u | wc -l
4
```

# Action

So, this begs the question, why doesn't `parallel::detectCores()` do this? I'm pretty sure this has been discussed somewhere before. The first task is to identify any discussions and rationales for the current implementation.

# See also

* This is related to Issue #47 (`detectThreadsPerCore()`)
* It's complicated, e.g. https://superuser.com/questions/378273/strange-cpuinfo-how-many-cores-does-these-physical-cpu-has/932418#932418
* R-devel thread 'Detecting physical CPUs in detectCores() on Linux platforms' started on 2023-08-07 ().

Contributor guide

Open the contributing guide

Research direction

Begin with parallel::detectCores() and the Linux /proc/cpuinfo commands documented in the issue; review Issue #47 and the linked R-devel thread for prior rationale. Done means documenting the existing constraints and reaching a clearly supported approach for distinguishing logical from physical CPUs on Linux.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.