h2oai / h2oai/datatable

fread in parallel mode, multi-core performance decreases, when the number of cores exceeds 20

Open
#2,262 3 comments 0 reactions 0 assignees View on GitHub
fread performance question
Dominant language
C++
Stars
1.9k
Forks
164
Avg merge
7h 31m
Merged PRs (30d)
1

Description

During the fread operation, the performance decreases significantly in the case of more than 20 cores. According to Amdahl's law, the higher the degree of parallelism, the performance may stabilize, but why is there a significant decrease? This will result in the default nthreads setting, on some machines with more than 20 cores, the best performance will not be achieved, but a poor performance.

**Machine environment: logical core 40 cores, physical core 20 cores, hyper-threading turned on**

data size is 10g, yellow_tripdata csv

[Performance data is the cumulative time of ten runs]
nthreads: 1, performance: 360.42344781407155
nthreads: 4, performance: 100.93926633079536
nthreads: 8, performance: 59.12695063999854
nthreads: 16, performance: 39.5492869140580
nthreads: 20, performance: 37.37294516689144
nthreads: 28, performance: 93.49912556796335
nthreads: 32, performance: 293.03483925387263
nthreads: 40, performance: slower than single thread

Machine environment: logical core 40 cores, physical core 20 cores, hyper-threading turned on

processor : 39
vendor_id : GenuineIntel
cpu family : 6
model : 79
model name : Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
stepping : 1
microcode : 0xb000036
cpu MHz : 2394.495
cache size : 25600 KB
physical id : 1
siblings : 20
core id : 12
cpu cores : 10
apicid : 57
initial apicid : 57
fpu : yes
fpu_exception : yes
cpuid level : 20
wp : yes

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.