mlcommons / mlcommons/storage

Accelerator's count and read_thread's count were used as criteria for sweep evaluation, but the results are confusing.

Open
#195 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
203
Forks
67
Avg merge
20m
Merged PRs (30d)
8

Description

I ran evaluations while varying the number of accelerators and the number of read threads. I expected the I/O throughput to scale proportionally with accelerator × read_thread, but as shown below the performance drops dramatically when I use five accelerators. Even if I keep increasing the number of accelerators and read threads, I thought the system would eventually hit a bottleneck, yet instead of a gradual bottleneck the performance collapses.

For the tests I only added the read_thread option to the default run command you provided, and before each run I re‑formatted the NVMe device and remounted the XFS filesystem (the initialization step).

Could you help me figure out why I’m getting these results? Is there something wrong with my setup?

Image

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are named. Start by reproducing the default run with read_thread, five accelerators, and the stated NVMe reformat/XFS remount initialization, then compare the sweep results across accelerator and read_thread counts. Done means identifying whether the collapse comes from the setup or benchmark behavior and documenting the cause.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.