mlcommons / mlcommons/endpoints-submission-cli

Separate 'run date' and 'run upload date'

Open
#76 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1
Forks
1
Avg merge
2d 13h
Merged PRs (30d)
6

Description

Summary

The submission CLI currently conflates two distinct concepts under a single "date": the date a benchmark run actually happened, and the date a run/submission was uploaded to the API. We should treat these as two separate fields with different semantics.

Background

Today, runs create and submissions create-local each compute a single timestamp for a run in different, inconsistent ways (one derives it from the benchmark's own log when available and falls back to "now"; the other always uses "now"). That timestamp is then used to populate the run_date field, which can end up not reflecting when the benchmark was actually executed.

Proposal

Introduce two clearly separate fields:

  1. Run date — the date the benchmark run actually started/completed. This must come from the benchmark's own output log (e.g. result_summary.json). It should never be inferred or defaulted by the CLI.
    • If the run log does not contain this information, the CLI should error out rather than silently substituting a fallback value (like the current upload time). This is a signal that the benchmark output itself is missing required data.
  2. Run upload date — the date the run/submission was uploaded via the CLI. This can simply be the current timestamp at upload time, since that's genuinely what it represents.

Notes

  • The exact field/schema for how "run date" is emitted by the benchmark log is still being finalized, so this issue is to track the requirement, not to implement it yet.
  • Once the benchmark output format is finalized, the CLI's run-parsing logic should be updated to read run date strictly from the log and fail fast if it's absent, and to record upload date as a separate, independent field.

This issue was created by Claude.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the runs create and submissions create-local command paths and their run-parsing logic. Wait for the benchmark output schema to be finalized, then verify that run date is read only from the log, upload date is recorded independently, and missing run-date data produces an error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.