Rules.md does not describe public_ids.json, which reportgen now writes into submission packages
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 203
- Forks
- 67
- Avg merge
- 20m
- Merged PRs (30d)
- 8
Description
Summary
reportgen now writes a public_ids.json registry beside the global results
rollup, and creates it automatically on any tree that lacks one. Rules.md does
not mention the file, and no submission check looks at it — so it will appear
inside submission packages silently. The WG should decide whether that file is
required, permitted, or forbidden in a submission, and Rules.md should say so.
What the file is
Public IDs used to be positional: results.csv is sorted, then numbered
v3.0-0001 upward on every run, so withholding or inserting a row renumbers
everything after it. That is harmless until reviewers, issues and assignment
documents cite IDs by number, at which point a stale citation does not look
broken — it points at a different, valid row. It bit the v3.0 tree twice in one
day (see #836): 172 of 177 rows renumbered, invalidating every ID cited in
ApparentProblems.md, AssignedReviewers.md and in review issues.
public_ids.json records identity -> Public ID, where identity is the
reportgen workload key plus benchmark type. A row keeps its ID however the table
re-sorts around it; a genuinely new row mints max + 1; a row that stops
publishing leaves its ID reserved and a gap in the table, and gets it back if the
submission is fixed. Deleting the file is what renumbers a tree — done once
before publication to close gaps and restore sort order.
Implemented in mlcommons/storage@8c9dd75 and mlcommons/storage@9acf602. Applied to
the v3.0 tree in mlcommons/submissions_storage_v3.0#40.
Why it needs a Rules.md decision
The file is created automatically — presence is not opt-in — so every
submitter who runs mlpstorage reports reportgen against their own results
directory gets one. Where it lands depends on the tree shape, and one of the two
cases is inside the submission:
| Tree shape | Registry path |
|---|---|
| Aggregated multi-org tree (staff) | <results-dir>/public_ids.json — repo root, outside any submission |
A submitter's own results/ folder |
<division>/<org>/results/public_ids.json — inside the submission package |
The second case is the ordinary submitter workflow, and nothing currently
notices the file:
2.1.5 requiredSubdirectoriesClosed/requiredSubdirectoriesOpenenumerate
entries directly inside<division>/<org>/, so a file one level down in
results/is out of scope.2.1.8 resultsDirectorySystemsenumerates<org>/results/withlist_dir(),
which filters to directories only, so a JSON file there is invisible to it.
Verified: copying public_ids.json into closed/<org>/results/ in the v3.0 tree
leaves mlpstorage validate byte-identical — 330 errors / 183 warnings / 254
informational, zero lines mentioning the file.
So today it ships, unmentioned and unchecked. That is the part worth deciding
deliberately rather than by default.
Options
- Permit and document it. Add it to the Rules.md §2.1 directory listings as
an optional generated file, alongside whatever is decided for the generated
rollups (see below). Submitters who have one keep it; nobody has to produce
one. - Require it. Makes IDs stable per submission from first generation. Costs
submitters nothing (it is generated), but makes a missing file a violation for
submissions produced by older tooling. - Forbid it in packages. Submitters delete it before packaging, and a check
flags it. Keeps packages to exactly what Rules.md describes, at the cost of a
step submitters will forget and a check that has to exist. - Do not write it in submitter mode. Restrict creation to the aggregated
staff tree. Cleanest for packages, but loses per-submission ID stability and
adds a mode distinction to reportgen that does not exist today.
Recommendation: option 1. The file is small, generated, and harmless where it
lands; and the alternative that keeps packages pristine (option 3) relies on a
manual deletion step, which is the kind of thing that silently does not happen.
Related gap worth folding in
Rules.md does not describe reportgen's generated rollups either. §2.1.16
(runResultsJson) and §2.1.22 (checkpointingResultsJson) require the
DLIO-produced results.json inside run and workload directories, and the §2.1
directory trees show those — but the results.csv / results.json rollups that
reportgen writes at the per-model, per-org and global levels appear nowhere, and
results.csv is not mentioned in Rules.md at all. Whoever picks this up should
probably settle both in one edit, since the answer for public_ids.json is
really "what does Rules.md say about generated files in a package".
Acceptance
- Rules.md states whether
public_ids.jsonis required, permitted or forbidden
in a submission package, and shows it in the §2.1 directory listings if
permitted or required. - The same for the generated
results.csv/results.jsonrollups. - If forbidden, a submission check enforces it; if required, a check enforces
that too. If merely permitted, no check is needed and the current silence is
correct.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with Rules.md §2.1 and the issue's four policy options, then inspect checks 2.1.5 and 2.1.8 to understand how submission contents are currently enumerated. Decide how public_ids.json and reportgen's results.csv/results.json rollups should be treated, update the directory listings and rules, and add enforcement only if the chosen policy requires it.
Written by the indexing model from the issue text.
Assessment
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100