Test fixtures for training & checkpointing use fabricated DLIO metric keys real DLIO never emits
@FileSystemGuy is already working on this.
Since Jul 21, 2026.
- Dominant language
- Python
- Stars
- 203
- Forks
- 67
- Avg merge
- 20m
- Merged PRs (30d)
- 8
Description
Summary
The training and checkpointing summary.json test fixtures use metric keys that real DLIO never emits. Tests therefore validate reportgen/aggregation against a metric shape that does not exist in production, and the _aggregate_* code paths grow columns that can never populate from a real run.
Evidence
Real DLIO output (dlio_benchmark/utils/statscounter.py, DLIO_local_changes):
- Training (
statscounter.py:201-205):train_throughput_samples_per_second— listtrain_io_mean_MB_per_second— scalar (np.mean(train_throughput)*record_size/1024/1024; logged "MiB/second" — theMBlabel is MiB)train_io_stdev_MB_per_second— scalar
- Checkpointing (
statscounter.py:178-185):save_checkpoint_io_mean_GB_per_second,save_checkpoint_duration_mean_seconds(+_stdev_) — scalarsload_checkpoint_io_mean_GB_per_second,load_checkpoint_duration_mean_seconds(+_stdev_) — scalars
What the fixtures contain instead:
- Training fixtures —
metrickeys:train_au_percentage,train_throughput_samples_per_second,train_io_throughput_MB_per_second(a list key that does not exist in real DLIO — real is the scalartrain_io_mean_MB_per_second). - Checkpointing fixtures —
metrickeys:checkpoint_read_throughput_GB_per_second,checkpoint_write_throughput_GB_per_second(list keys; real DLIO emitssave/loadscalar means + duration means, with noread/writethroughput lists).
Affected fixtures/helpers (non-exhaustive):
tests/fixtures/sample_results/training/**/summary.json(andmulti_orgname/,degenerate/trees)tests/fixtures/sample_results/checkpointing/**/summary.jsontests/conftest.pycheckpointing fixture builder- referenced in
tests/unit/test_aggregation.py,tests/unit/test_rules_extractors.py
Impact
- Coverage gap. The reportgen aggregation and rules extractors are exercised against a metric layout production never produces, so bugs specific to the real DLIO key names/shapes can slip through.
- Dead aggregation columns.
_aggregate_training/_aggregate_checkpointingbuildtrain_mean_of_*/checkpoint_mean_of_*columns by iterating the list metric keys present. Against the fabricated fixtures this yieldstrain_mean_of_io_throughput_MB_per_secondandcheckpoint_mean_of_read/write_throughput_GB_per_second— columns that can never populate from a real run because the source keys don't exist. - Trap for future work. The reportgen v3.0 final-table slices had to source the real metrics (Read B/W, checkpoint bw/durations) from the real scalar keys and ship self-contained realistic
summary.jsonin their new tests, precisely because the shared fixtures were unrealistic. Anyone reusing the shared fixtures inherits the mismatch.
Note: the fixed webpage-parity
results.{csv,json}schema no longer emits the dead*_mean_of_*columns, so this is not a user-visible output bug today — it is test-fidelity / tech-debt that should be corrected so fixtures reflect real DLIO.
Proposed fix
Regenerate the training & checkpointing summary.json fixtures (and the conftest.py builder) to use the real DLIO metric keys:
- Training: replace the fabricated
train_io_throughput_MB_per_secondlist with the real scalarstrain_io_mean_MB_per_second/train_io_stdev_MB_per_second, keeping the realtrain_throughput_samples_per_secondlist andtrain_au_percentage. - Checkpointing: replace
checkpoint_{read,write}_throughput_GB_per_secondwith the realsave/loadscalars*_io_mean_GB_per_secondand*_duration_mean_seconds(+ stdevs).
Update test_aggregation.py / test_rules_extractors.py expectations accordingly. Decide whether to keep the *_mean_of_* aggregation path at all (it aggregates list metrics; with the real scalar-only checkpoint/IO keys there are fewer list metrics to aggregate).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.