reportgen silently drops kv_cache/vdb runs missing *_metadata.json — metrics publish as blank

Summary

When a KVCache or VectorDB run leaf is missing its *_metadata.json file, the entire run is silently dropped from the report. The submitter's measured data is present and well-formed on disk, but every metric column for that workload comes out blank. The only trace is a WARNING line that names neither the consequence nor the missing file.

This was found while regenerating results.csv for the v3.0 submission tree: one submitter's KVCache columns published as blank even though all of their run data was intact.

Mechanism

ResultFilesExtractor.extract (mlpstorage_py/rules/models.py:976-986) resolves the benchmark type in two steps:

1. Load *_metadata.json; if present and complete, build from it. This is the only path that identifies KVCache and VectorDB runs.
2. Otherwise fall through to DLIOResultParser.parse, which infers the type from a DLIO workflow signal or Hydra configs (models.py:862-893).

A KVCache run has neither signal, so step 2 raises:

ValueError: Could not determine benchmark type for <dir>:
summary.json lacks workflow signal and no Hydra configs found

get_runs_files catches that and logs Failed to load run from {root}: {e} (mlpstorage_py/rules/utils.py:479-481), then continues. The run leaf never enters the report. Where a datasize/datagen sibling leaf still loads, the workload keeps a row in the table but with no metrics — so the output looks like a submitter with no results, not like a tooling failure.

DLIO training and checkpointing runs are unaffected: their Hydra configs still identify them, so the metadata file is effectively optional for those.

Reproducer

# Any valid kv_cache run directory
mv run/<ts>/kv_cache_<ts>_metadata.json /tmp/
mlpstorage reports reportgen --results-dir <tree>
# -> WARNING: Failed to load run from .../run/<ts>: Could not determine benchmark type
# -> results.csv: every KVCache metric column for that system is empty

Confirmed against a real submission: 5 run leaves affected, each with a well-formed summary.json (options 1/2/3 present) and all 1,836 per-rank kvcache_results_*.json files intact.

Why the current warning is not enough

The message says a run failed to load. It does not say:

• that the run's data will be omitted from the results table — the actual user-visible consequence;
• that the specific missing input is *_metadata.json;
• that this is recoverable by restoring one small file rather than rerunning the benchmark.

At scale the line is easy to miss — the run that surfaced this produced ~15,000 lines of reportgen output, in which these were 5 warnings among many.

Suggested fix

Keeping the strict behavior is defensible: arguably a submission missing *_metadata.json is incomplete, and inferring the benchmark type from summary.json shape risks masking genuinely malformed packages. But the failure should be loud and actionable. Suggestions, roughly in order of value:

1. Name the consequence and the cause. When a non-DLIO leaf fails type detection and a summary.json is present, warn along the lines of: Dropping <dir> from the report: no *_metadata.json and no DLIO workflow signal. This workload's metrics will be BLANK in results.csv. If this is a kv_cache/vector_database run, restore its <type>_<ts>_metadata.json.
2. Summarize dropped runs at the end of the reportgen pass, so the count is visible without grepping the log.
3. Optionally, have the submission checker flag a run leaf that has summary.json but no *_metadata.json, so it is caught at validation time rather than at report time.
4. Optionally, infer the type from the unmistakable KVCache summary shape (schema_version + options + trials_per_option) as a last-resort fallback that warns loudly when it fires. Listed last deliberately — the working group may prefer submitters to supply the real file.

Environment

mlpstorage_py on reportgen-column-parity; line references from that branch.

Start with ResultFilesExtractor.extract in mlpstorage_py/rules/models.py:976-986 and the exception handling in mlpstorage_py/rules/utils.py:479-481. Reproduce the missing-metadata case with reportgen, then inspect the existing warning path and report-generation flow. Done means the affected run's omission and missing *_metadata.json are clearly reported, with any agreed dropped-run summary covered by tests or verification.