mlcommons / mlcommons/storage

Rules.md does not describe public_ids.json, which reportgen now writes into submission packages

Open
#838 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Future mlpstorage Rules
Dominant language
Python
Stars
203
Forks
67
Avg merge
20m
Merged PRs (30d)
8

Description

Summary

reportgen now writes a public_ids.json registry beside the global results
rollup, and creates it automatically on any tree that lacks one. Rules.md does
not mention the file, and no submission check looks at it — so it will appear
inside submission packages silently. The WG should decide whether that file is
required, permitted, or forbidden in a submission, and Rules.md should say so.

What the file is

Public IDs used to be positional: results.csv is sorted, then numbered
v3.0-0001 upward on every run, so withholding or inserting a row renumbers
everything after it. That is harmless until reviewers, issues and assignment
documents cite IDs by number, at which point a stale citation does not look
broken — it points at a different, valid row. It bit the v3.0 tree twice in one
day (see #836): 172 of 177 rows renumbered, invalidating every ID cited in
ApparentProblems.md, AssignedReviewers.md and in review issues.

public_ids.json records identity -> Public ID, where identity is the
reportgen workload key plus benchmark type. A row keeps its ID however the table
re-sorts around it; a genuinely new row mints max + 1; a row that stops
publishing leaves its ID reserved and a gap in the table, and gets it back if the
submission is fixed. Deleting the file is what renumbers a tree — done once
before publication to close gaps and restore sort order.

Implemented in mlcommons/storage@8c9dd75 and mlcommons/storage@9acf602. Applied to
the v3.0 tree in mlcommons/submissions_storage_v3.0#40.

Why it needs a Rules.md decision

The file is created automatically — presence is not opt-in — so every
submitter who runs mlpstorage reports reportgen against their own results
directory gets one. Where it lands depends on the tree shape, and one of the two
cases is inside the submission:

Tree shape Registry path
Aggregated multi-org tree (staff) <results-dir>/public_ids.json — repo root, outside any submission
A submitter's own results/ folder <division>/<org>/results/public_ids.json — inside the submission package

The second case is the ordinary submitter workflow, and nothing currently
notices the file:

  • 2.1.5 requiredSubdirectoriesClosed / requiredSubdirectoriesOpen enumerate
    entries directly inside <division>/<org>/, so a file one level down in
    results/ is out of scope.
  • 2.1.8 resultsDirectorySystems enumerates <org>/results/ with list_dir(),
    which filters to directories only, so a JSON file there is invisible to it.

Verified: copying public_ids.json into closed/<org>/results/ in the v3.0 tree
leaves mlpstorage validate byte-identical — 330 errors / 183 warnings / 254
informational, zero lines mentioning the file.

So today it ships, unmentioned and unchecked. That is the part worth deciding
deliberately rather than by default.

Options

  1. Permit and document it. Add it to the Rules.md §2.1 directory listings as
    an optional generated file, alongside whatever is decided for the generated
    rollups (see below). Submitters who have one keep it; nobody has to produce
    one.
  2. Require it. Makes IDs stable per submission from first generation. Costs
    submitters nothing (it is generated), but makes a missing file a violation for
    submissions produced by older tooling.
  3. Forbid it in packages. Submitters delete it before packaging, and a check
    flags it. Keeps packages to exactly what Rules.md describes, at the cost of a
    step submitters will forget and a check that has to exist.
  4. Do not write it in submitter mode. Restrict creation to the aggregated
    staff tree. Cleanest for packages, but loses per-submission ID stability and
    adds a mode distinction to reportgen that does not exist today.

Recommendation: option 1. The file is small, generated, and harmless where it
lands; and the alternative that keeps packages pristine (option 3) relies on a
manual deletion step, which is the kind of thing that silently does not happen.

Related gap worth folding in

Rules.md does not describe reportgen's generated rollups either. §2.1.16
(runResultsJson) and §2.1.22 (checkpointingResultsJson) require the
DLIO-produced results.json inside run and workload directories, and the §2.1
directory trees show those — but the results.csv / results.json rollups that
reportgen writes at the per-model, per-org and global levels appear nowhere, and
results.csv is not mentioned in Rules.md at all. Whoever picks this up should
probably settle both in one edit, since the answer for public_ids.json is
really "what does Rules.md say about generated files in a package".

Acceptance

  • Rules.md states whether public_ids.json is required, permitted or forbidden
    in a submission package, and shows it in the §2.1 directory listings if
    permitted or required.
  • The same for the generated results.csv / results.json rollups.
  • If forbidden, a submission check enforces it; if required, a check enforces
    that too. If merely permitted, no check is needed and the current silence is
    correct.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with Rules.md §2.1 and the issue's four policy options, then inspect checks 2.1.5 and 2.1.8 to understand how submission contents are currently enumerated. Decide how public_ids.json and reportgen's results.csv/results.json rollups should be treated, update the directory listings and rules, and add enforcement only if the chosen policy requires it.

Written by the indexing model from the issue text.

Assessment

Domain
documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.