mlcommons / mlcommons/storage

mlpstorage incorrectly identifies llama3-8b on 16 GPU as qualified CLOSED submission

Open
#170 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

DLIO or mlpstorage
Dominant language
Python
Stars
203
Forks
67
Avg merge
20m
Merged PRs (30d)
8

Description

See the log below. It should be set as OPEN

2025-06-20 09:55:15|STATUS: Benchmark results directory: ./results/eagle/n2x8/checkpointing/llama3-8b/20250620_095514
2025-06-20 09:55:15|INFO: Found benchmark run: checkpointing_run_llama3-8b_20250620_095514
2025-06-20 09:55:15|STATUS: Verifying benchmark run for checkpointing_run_llama3-8b_20250620_095514
2025-06-20 09:55:15|STATUS: Benchmark run qualifies for CLOSED category ([RunID(program='checkpointing', command='run', model='llama3-8b', run_datetime='20250620_095514')])
2025-06-20 09:55:15|WARNING: Running the benchmark without verification for open or closed configurations. These results are not valid for submission. Use --open or --closed to specify a configuration.
2025-06-20 09:55:15|STATUS: Instantiated the Checkpointing Benchmark...
2025-06-20 09:55:15|STATUS: Running benchmark command:: mpiexec -n 16 --ppn 8 --cpu-bind depth -d 16 /home/hzheng/crux/DLIO/dlio_benchmark/workspace/2025-06-19/pyenvs/2025-06-19/storage/bin/dlio_benchmark workload=llama3_8b ++hydra.run.dir=./results/eagle/n2x8/checkpointing/llama3-8b/20250620_095514 ++hydra.output_subdir=dlio_config ++workload.checkpoint.num_checkpoints_read=0 ++workload.checkpoint.num_checkpoints_write=10 ++workload.checkpoint.checkpoint_folder=.//checkpoints/n2x8/llama3-8b --config-dir=/lus/eagle/projects/PolarisAT/hzheng/crux/DLIO/dlio_benchmark/workspace/2025-06-19/storage/configs/dlio

@wvaske Could you please address this?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the verification and qualification path that emits the logged “qualifies for CLOSED category” message, using the llama3-8b 16-GPU run shown here as the reproduction case. Trace why it selects CLOSED and verify that this case is classified as OPEN; the issue provides no file or test name to narrow the search.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.