mlcommons / mlcommons/inference

1633 ERRORS encountered when running "sh ./reference_mlperf_accuracy.sh"

Open
#2,310 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.6k
Forks
650
Avg merge
1d 22h
Merged PRs (30d)
6

Description

================================================
SUT name : PySUT
Scenario : Offline
Mode : PerformanceOnly
Samples per second: 0.857965
Tokens per second: 289.078
Result is : VALID
Min duration satisfied : Yes
Min queries satisfied : Yes
Early stopping satisfied: Yes

================================================
Additional Stats

Min latency (ns) : 2785715494
Max latency (ns) : 1903340943861
Mean latency (ns) : 955159206269
50.00 percentile latency (ns) : 961910964272
90.00 percentile latency (ns) : 1712469406533
95.00 percentile latency (ns) : 1809927827424
97.00 percentile latency (ns) : 1847756820559
99.00 percentile latency (ns) : 1884148081334
99.90 percentile latency (ns) : 1902868349502

================================================
Test Parameters Used

samples_per_query : 1633
target_qps : 1
ttft_latency (ns): 100000000
tpot_latency (ns): 100000000
max_async_queries : 1
min_duration (ms): 600000
max_duration (ms): 0
min_query_count : 1
max_query_count : 0
qsl_rng_seed : 1780908523862526354
sample_index_rng_seed : 14771362308971278857
schedule_rng_seed : 18209322760996052031
accuracy_log_rng_seed : 0
accuracy_log_probability : 0
accuracy_log_sampling_target : 0
print_timestamps : 0
performance_issue_unique : 0
performance_issue_same : 0
performance_issue_same_index : 0
performance_sample_count : 1633
WARNING: sample_concatenate_permutation was set to true.
Generated samples per query might be different as the one in the setting.
Check the generated_samples_per_query line in the detailed log for the real
samples_per_query value

No warnings encountered during test.

1633 ERRORS encountered. See detailed log.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by rerunning sh ./reference_mlperf_accuracy.sh and inspect its detailed log, since the report only summarizes 1633 errors despite a VALID result. Compare the detailed errors with the PySUT, Offline, and PerformanceOnly settings shown here; done means identifying the cause and making the accuracy run complete without those errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, shell
Domain
machine-learning, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.