reportgen: non-canonical kv_cache run layout publishes one run's metrics and silently discards the rest

Summary

When a kv_cache workload directory contains multiple run directories that do not match the canonical <model>/run/<timestamp>/ layout, reportgen publishes a results row built from one of those runs and silently discards the rest. The only diagnostic emitted is No valid run directories found, which describes neither the row that was produced nor the measurements that were dropped.

This is worse than dropping the workload entirely: a blank row is visible to a reviewer, whereas a populated row assembled from one of several measured configurations is not.

Observed in

mlcommons/submissions_storage_v3.0 @ 316717f2e, ANL crux-eagle, tool version 3.0.46.

The submission organizes its kv_cache runs by cluster topology instead of by run timestamp:

closed/ANL/results/crux-eagle/kv_cache/llama3-8b-10u/
├── 1nodex8ppn/    (8 ranks)    summary.json + kv_cache_20260723_062638_metadata.json
├── 8nodex8ppn/    (64 ranks)   summary.json + kv_cache_20260723_191548_metadata.json
└── 64nodex8ppn/   (512 ranks)  summary.json + kv_cache_20260724_051511_metadata.json

All three directories are complete: each has a summary.json with options 1/2/3, option subdirectories, a *_metadata.json, and 389 per-rank result files between them. Nothing is missing from the submission.

Behavior

All three runs are discovered and verified. From the reportgen log:

INFO: Created benchmark run: kv_cache_run_llama3.1-8b_20260723_062638
INFO: Created benchmark run: kv_cache_run_llama3.1-8b_20260724_051511
INFO: Created benchmark run: kv_cache_run_llama3.1-8b_20260723_191548
STATUS: Benchmark run qualifies for CLOSED category (... run_datetime='20260723_062638')
STATUS: Benchmark run qualifies for CLOSED category (... run_datetime='20260724_051511')
STATUS: Benchmark run qualifies for CLOSED category (... run_datetime='20260723_191548')
WARNING: No valid run directories found in .../crux-eagle/kv_cache/llama3-8b-10u

The resulting tables disagree with each other:

Table	Rows
crux-eagle/kv_cache/llama3-8b-10u/results.csv	0 (and results.json is [])
crux-eagle/kv_cache/results.csv	1
top-level results.csv	1 (v3.0-0005)

The workload-level table is empty, consistent with the warning. But the system rollup and the global table both carry a row — and its values are 1nodex8ppn's alone:

Cell	Published in v3.0-0005	1nodex8ppn	8nodex8ppn	64nodex8ppn
8b Storage Only tok/s	5865.99	5865.99	48.57	0.0
8b Storage + Mem tok/s	49.84	49.84	170.16	163.96
70b Storage Only tok/s	2521.63	2521.63	1110.03	738.48
8b Storage Only Read B/W	0.0	0.0	0.776	0.0
# Client Nodes	1	1	8	64

The 8-node and 64-node measurements — which have materially different bandwidth and latency figures — do not appear anywhere in any published table. The three zero-valued bandwidth cells and three blank P95 cells are likewise the 1-node run's own values, not placeholders.

Empirically the earliest-timestamped run wins, but I did not pin the selection rule.

Apparent root cause

Two different paths disagree about what constitutes a run:

• Row placement uses the canonical layout. _validate_model_dir (mlpstorage_py/reporting/directory_validator.py:411) accepts a child of the model directory only if _is_datetime_dir(entry.name) or entry.name in EXPECTED_COMMANDS. 1nodex8ppn is neither, so nothing is placed into the workload-level table and the warning at line 428 fires.
• Rollup aggregation consumes the extracted BenchmarkRun objects directly. Those were built successfully — each topology directory has a *_metadata.json, so ResultFilesExtractor.extract (mlpstorage_py/rules/models.py:976) identifies all three as kvcache runs without needing the directory name.

So the runs exist in memory, are excluded from the leaf table, and then collapse into a single row upstream because all three share the same (program, command, model) grouping key and differ only in run_datetime.

Why the warning is inadequate

No valid run directories found is emitted against the workload directory, is a WARNING among 183 others, and names no consequence. Nothing in the output states that a row was published, that it represents one of three configurations, or which two were dropped. In a 179-row table this is not something a reviewer can be expected to catch — the row looks complete.

Suggested fixes

Independent of any decision about whether the layout itself should be accepted:

1. Never silently reduce N measured configurations to one. If multiple runs group to one row and their metrics disagree, that is either an error or a row per configuration — not a silent pick.
2. Make the warning name the consequence, in the style of #835: say that a row was published from a subset, how many runs were dropped, and which.
3. Make the two paths agree. A run good enough to extract, verify and aggregate into a rollup should not be invisible to the leaf table that is supposed to contain it — the disagreement between an empty llama3-8b-10u/results.csv and a populated parent rollup is itself a bug.
4. Submission-checker gate: a workload whose leaf table has zero rows while its rollup has one should fail validation.

Item 1 is the one that affects published results.

Related

• #835 — silent drop of kv_cache/VDB run leaves missing *_metadata.json. Same family of problem (a run disappears without the output saying so), different trigger. This one is more severe because it produces a wrong number rather than a blank.
• The layout question itself is a separate WG matter, written up as open question 3 in ApparentProblems.md.

Start with _validate_model_dir in mlpstorage_py/reporting/directory_validator.py around line 411 and ResultFilesExtractor.extract in mlpstorage_py/rules/models.py around line 976. Trace how non-canonical kv_cache directories reach rollup aggregation but not the workload-level table. Done means conflicting runs are not silently reduced to one published row, the leaf and rollup paths agree, and diagnostics identify any dropped runs and their consequence.