Endpoints run output vs. the submission CLI

Short answer: a report directory produced by endpoints cannot be handed to
endpoints-submission-cli as-is. The two tools agree on the contents of the
performance numbers (result_summary.json is literally the endpoints Report
schema), but they disagree on file names, directory depth, and on several files
that endpoints never writes at all.

Analysed on 2026-08-31 against the checkouts in endpoints/ and
endpoints-submission-cli/.

---

1. What an endpoints run actually writes

A single run — performance phase and accuracy phase in the same config, e.g.
examples/11_Edge_Agentic_Example/online_edge_full_run.yaml
— lands everything in one report_dir:

<report_dir>/
├── config.yaml                     resolved config, secrets redacted
├── report.txt                      human-readable dump of the run
├── events.jsonl                    one line per request (prompts, responses, errors)
├── sample_idx_map.json             {dataset: {uuid -> sample index}}
├── performance/
│   └── result_summary.json         qps, tps, ttft/tpot/latency percentiles, ISL/OSL,
│                                   n_samples_issued/completed/failed, duration_ns,
│                                   run_config (load pattern, warmup, seeds), git_sha
├── accuracy/
│   └── accuracy_results.json       {"average_accuracy": …, "accuracy_scores": [ … ]}
├── metrics/
│   ├── final_snapshot.json
│   └── .ready
├── scores.json                     only for the inline agentic checker
├── profiling.json                  only when profiling is enabled
├── audit/<phase>/                  only when an `audit:` block is configured
└── plots/                          only when you run scripts/plot_results.py

Notes worth remembering:

• performance/ and report.txt are always written. accuracy/ appears only
  if something was scored — a perf-only run leaves no accuracy/ folder
  (accuracy.py:386).
• Accuracy is deliberately not duplicated into result_summary.json; the
  serializer pops it
  (report.py:393).
• report_dir defaults to a timestamped dir under /tmp when the config
  doesn't set one.
• One report directory = one run = one concurrency level. A Pareto curve is N
  separate report directories.

---

2. What the submission CLI expects

Three layers, each with its own expectations.

2a. runs create --path <folder> — the run folder

From runs/parser.py:

<run_folder>/
├── system_desc.json        REQUIRED — org / system / model / dataset metadata
├── config.yaml             REQUIRED
├── result_summary.json     REQUIRED — at the top level, not under performance/
├── run_metadata.json       headline metrics + latency table (used downstream)
├── results.json            high-level summary; may carry accuracy_scores
├── accuracy/results.json   alternative home for accuracy
├── events.jsonl, report.txt, sample_idx_map.json, metrics/final_snapshot.json
├── serving_config.json     optional
├── src/<impl>/README.md    REQUIRED downstream — how to rebuild the SUT
└── documentation/          becomes the submission's docs/

The folder is tarred and uploaded whole, so anything extra rides along.

The published doc
docs/endpoints-cli/usage/runs.md
still says system_info.json and runtime_settings.json. The code says
system_desc.json; the doc is stale.

2b. submissions create — the assembled tree

submissions/builder.py
turns N run archives into:

<org>/<submission_id>/
├── src/<implementation>/README.md
├── docs/
└── results/<system>/
    ├── system_desc_id.json
    └── <model>/
        └── r<concurrency>/
            ├── point.yaml
            ├── result_summary.json
            ├── accuracy_results.json
            └── run_metadata.json

Two things drive the grouping:

• Concurrency comes from settings.load_pattern.target_concurrency →
  directory name r<N>.
• Run type comes from datasets[0].type — the first dataset only. A run
  is either "performance" or "accuracy", never both.

2c. submission-checker check — validation

Reads the tree above and enforces ~40 rules (point count 7–32, regional
coverage, seed = 42, stream_all_chunks: true, accuracy gate, tps
consistency, …).

The rule table in the CLI's
README.md describes an older
pareto/<system>/<model>/points/… layout with
mlperf_endpoints_log_summary.json. The code and the fixtures in
test_submissions/ both use results/<system>/<model>/r<N>/. Trust the code.

---

3. Where the two sides don't meet

#	What breaks	endpoints writes	CLI wants
1	Registration fails immediately	performance/result_summary.json	result_summary.json at the folder root
2	Registration fails immediately	nothing	system_desc.json (org, system name, max_supported_concurrency, model, dataset, accuracy target…)
3	Accuracy is silently dropped	accuracy/accuracy_results.json	accuracy/results.json or results.json
4	Accuracy shape mismatch	accuracy_scores is a list of dataset entries	a dict keyed by dataset name
5	Combined runs lose their accuracy	perf + accuracy in one report dir	a run is classified by datasets[0].type; a combined config reads as performance-only, so its accuracy is never written into the point
6	Checker error: run-metadata-present	nothing	run_metadata.json with ~40 non-null fields (system_tps, tps_per_user, tps_utilization, full ttft/tpot/request latency tables)
7	Checker error: point-config-valid	settings.warmup = {enabled, n_requests, salt, drain, warmup_random_seed}	warmup = {duration_s, requests_issued, requests_completed, data_source, concurrency, initialization_steps}, all non-null
8	Checker error: point-config-valid	settings.runtime.min_issue_duration_ms	min_duration_ms (the builder looks up the wrong key and writes null)
9	Checker error: streaming-config	stream_all_chunks defaults to false	must be true
10	Build fails: src-dir / src-readme	nothing	src/<impl>/README.md inside every run folder
11	Structural	one report dir per run	7–32 report dirs, one per concurrency, all registered separately

Verified by feeding an endpoints-shaped directory to the real parser:

$ parse_run_folder(<endpoints report_dir>)
RunFolderError: missing required file(s): system_desc.json, result_summary.json

Note also that endpoints ships its own
scripts/publish_submission.py, but it
targets a third layout — the upstream mlcommons/inference submission
checker ({division}/{submitter}/results/…/performance/run_1/). It is not a
bridge to this CLI.

---

4. What it would take to bridge

Nothing here is deep — the measured data all exists, it is packaged differently.
Roughly in order of effort:

1. A packaging step (a script on either side) that, per run, produces a flat
   folder: copy performance/result_summary.json → result_summary.json,
   config.yaml, events.jsonl, report.txt, metrics/final_snapshot.json,
   and rewrite accuracy/accuracy_results.json from a list into the
   dataset-keyed dict as accuracy/results.json.
2. Submitter-supplied metadata: system_desc.json and src/<impl>/README.md
   have no source in a benchmark run — they need a template plus a fill-in step.
3. run_metadata.json: fully derivable from result_summary.json
   (system_tps = total output tokens / duration, tps_per_user = that over
   concurrency, latency tables from the percentile blocks). tps_utilization is
   filled in by the builder across the curve, so a per-run placeholder is fine.
4. Warmup disclosure: either extend the endpoints WarmupConfig to record
   duration_s, requests_issued, requests_completed, data_source and
   initialization_steps during the warmup phase, or have the packaging step
   ask for them.
5. Two small CLI fixes, independent of endpoints: read
   min_issue_duration_ms in _write_point_yaml, and let a run contribute both
   performance and accuracy instead of picking a type from datasets[0].
6. Docs: the run-folder layout in docs/endpoints-cli/usage/runs.md and the
   directory tree in the CLI README.md both describe layouts the code no
   longer uses.

The cleanest split is (1)+(3) as an inference-endpoint subcommand — say
inference-endpoint submission package — since everything it needs is already
in the report directory, with (2) and (4) prompted or read from a small
submitter file.

Start with src/endpoints_submission_cli/runs/parser.py and submissions/builder.py, then inspect the layouts in test_submissions/ and the documented paths in docs/endpoints-cli/usage/runs.md and README.md. Define the packaging and metadata inputs before changing behavior; done means an endpoints report can be registered, assembled into the current results tree, and validated by the submission checker without losing accuracy or required metadata.