mlcommons / mlcommons/storage

I was wondering if the cpu-bind option is released or if there are cases where the numa-core allocation may not be done properly.

Open
#201 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
203
Forks
67
Avg merge
20m
Merged PRs (30d)
8

Description

The run benchmark results in my test environment are much lower than I expected, and even if I test with several parameters sweep, all the AU expectations are showing up "Fail".
I wonder if the cpu-bind option is not set. (If you don't agree with my estimate, please comment.)

1. Is there a way to know if cpu-bind option is properly set in MLPerf Storage tool?

2. Which option of the reference links below should I use to run with cpu-bind set up properly? There are several options in the mix, so I can't figure out which option is the right one that is currently implemented.

Below is the issue link I referred to :
a. github.com/mlcommons/storage/issues/55: --cpu-bind depth -d num_workers
b. github.com/mlcommons/storage/issues/123
: --mpi-params --bind-to-core --cpu-per-proc 8 ? mpiexec --cpu-bind depth -d 4 ?

3. As shown below, some parameter sweep options seem to have a big difference between au percentage and thruhgput during 5 repetitive evaluations, can you advise what the problem is? Is the pcie generation link unraveling, or is there a problem with cpu-bind falling?

Image

4. Is there no problem with the warning log below? The actual number of my cores checked through the nproc command is 512 (Hyperthread on). DLIO seems to recognize only the threads per core value or Invalid value of unknown source.

[WARNING] Running DLIO with 16 threads for I/O but core available 2 are insufficient and can lead to lower performance.

Thank you.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source files, tests, or entry points are identified. Start by reproducing the benchmark with the reported cpu-bind options and inspecting how MLPerf Storage invokes DLIO and reports available cores; done means explaining the binding behavior, the warning, and the performance discrepancy or recording a confirmed bug.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.