mlcommons / mlcommons/storage

Test fixtures for training & checkpointing use fabricated DLIO metric keys real DLIO never emits

Open
#830 0 comments 0 reactions 1 assignee View on GitHub

@FileSystemGuy is already working on this.

Since Jul 21, 2026.

Future mlpstorage
Dominant language
Python
Stars
203
Forks
67
Avg merge
20m
Merged PRs (30d)
8

Description

Summary

The training and checkpointing summary.json test fixtures use metric keys that real DLIO never emits. Tests therefore validate reportgen/aggregation against a metric shape that does not exist in production, and the _aggregate_* code paths grow columns that can never populate from a real run.

Evidence

Real DLIO output (dlio_benchmark/utils/statscounter.py, DLIO_local_changes):

  • Training (statscounter.py:201-205):
    • train_throughput_samples_per_secondlist
    • train_io_mean_MB_per_secondscalar (np.mean(train_throughput)*record_size/1024/1024; logged "MiB/second" — the MB label is MiB)
    • train_io_stdev_MB_per_second — scalar
  • Checkpointing (statscounter.py:178-185):
    • save_checkpoint_io_mean_GB_per_second, save_checkpoint_duration_mean_seconds (+ _stdev_) — scalars
    • load_checkpoint_io_mean_GB_per_second, load_checkpoint_duration_mean_seconds (+ _stdev_) — scalars

What the fixtures contain instead:

  • Training fixtures — metric keys: train_au_percentage, train_throughput_samples_per_second, train_io_throughput_MB_per_second (a list key that does not exist in real DLIO — real is the scalar train_io_mean_MB_per_second).
  • Checkpointing fixtures — metric keys: checkpoint_read_throughput_GB_per_second, checkpoint_write_throughput_GB_per_second (list keys; real DLIO emits save/load scalar means + duration means, with no read/write throughput lists).

Affected fixtures/helpers (non-exhaustive):

  • tests/fixtures/sample_results/training/**/summary.json (and multi_orgname/, degenerate/ trees)
  • tests/fixtures/sample_results/checkpointing/**/summary.json
  • tests/conftest.py checkpointing fixture builder
  • referenced in tests/unit/test_aggregation.py, tests/unit/test_rules_extractors.py

Impact

  1. Coverage gap. The reportgen aggregation and rules extractors are exercised against a metric layout production never produces, so bugs specific to the real DLIO key names/shapes can slip through.
  2. Dead aggregation columns. _aggregate_training / _aggregate_checkpointing build train_mean_of_* / checkpoint_mean_of_* columns by iterating the list metric keys present. Against the fabricated fixtures this yields train_mean_of_io_throughput_MB_per_second and checkpoint_mean_of_read/write_throughput_GB_per_second — columns that can never populate from a real run because the source keys don't exist.
  3. Trap for future work. The reportgen v3.0 final-table slices had to source the real metrics (Read B/W, checkpoint bw/durations) from the real scalar keys and ship self-contained realistic summary.json in their new tests, precisely because the shared fixtures were unrealistic. Anyone reusing the shared fixtures inherits the mismatch.

Note: the fixed webpage-parity results.{csv,json} schema no longer emits the dead *_mean_of_* columns, so this is not a user-visible output bug today — it is test-fidelity / tech-debt that should be corrected so fixtures reflect real DLIO.

Proposed fix

Regenerate the training & checkpointing summary.json fixtures (and the conftest.py builder) to use the real DLIO metric keys:

  • Training: replace the fabricated train_io_throughput_MB_per_second list with the real scalars train_io_mean_MB_per_second / train_io_stdev_MB_per_second, keeping the real train_throughput_samples_per_second list and train_au_percentage.
  • Checkpointing: replace checkpoint_{read,write}_throughput_GB_per_second with the real save/load scalars *_io_mean_GB_per_second and *_duration_mean_seconds (+ stdevs).

Update test_aggregation.py / test_rules_extractors.py expectations accordingly. Decide whether to keep the *_mean_of_* aggregation path at all (it aggregates list metrics; with the real scalar-only checkpoint/IO keys there are fewer list metrics to aggregate).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.