mlcommons / mlcommons/storage

The problem that the reader.read_thread parameter does not utilize as much as the actual number of nproc.

Open
#205 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
203
Forks
67
Avg merge
20m
Merged PRs (30d)
8

Description

My CPU's thread count is 512 (I checked through the nproc command),
: 512 = 128core x 2 sockets x2(HyperThreading on)
It seems that this tool does not utilize as much as the actual number of nproc.
As you can see in the capture below, reader.read_threads parameter can only be allocated up to 128.

I checked again, and I think it only sweeps up to 128 threads in any environment. I don't know if it's a bottleleneck problem due to DLIO.
However, according to the run parameter guide you told me in #202, I think the total number of cores from the nproc command should be allocated to read_thread.

Could you check this problem?

Image

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the run parameter guide referenced in #202 and reproduce the reported limit for reader.read_threads on a system where nproc reports more than 128. Trace where the parameter is bounded and compare that behavior with the available processor count. Done means the intended relationship between nproc and read_thread is documented or corrected and verified on a high-thread-count system.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.