The problem that the reader.read_thread parameter does not utilize as much as the actual number of nproc.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 203
- Forks
- 67
- Avg merge
- 20m
- Merged PRs (30d)
- 8
Description
My CPU's thread count is 512 (I checked through the nproc command),
: 512 = 128core x 2 sockets x2(HyperThreading on)
It seems that this tool does not utilize as much as the actual number of nproc.
As you can see in the capture below, reader.read_threads parameter can only be allocated up to 128.
I checked again, and I think it only sweeps up to 128 threads in any environment. I don't know if it's a bottleleneck problem due to DLIO.
However, according to the run parameter guide you told me in #202, I think the total number of cores from the nproc command should be allocated to read_thread.
Could you check this problem?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the run parameter guide referenced in #202 and reproduce the reported limit for reader.read_threads on a system where nproc reports more than 128. Trace where the parameter is bounded and compare that behavior with the available processor count. Done means the intended relationship between nproc and read_thread is documented or corrected and verified on a high-thread-count system.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100