mlcommons / mlcommons/inference
1633 ERRORS encountered when running "sh ./reference_mlperf_accuracy.sh"
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 650
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 6
Description
================================================
SUT name : PySUT
Scenario : Offline
Mode : PerformanceOnly
Samples per second: 0.857965
Tokens per second: 289.078
Result is : VALID
Min duration satisfied : Yes
Min queries satisfied : Yes
Early stopping satisfied: Yes
================================================
Additional Stats
Min latency (ns) : 2785715494
Max latency (ns) : 1903340943861
Mean latency (ns) : 955159206269
50.00 percentile latency (ns) : 961910964272
90.00 percentile latency (ns) : 1712469406533
95.00 percentile latency (ns) : 1809927827424
97.00 percentile latency (ns) : 1847756820559
99.00 percentile latency (ns) : 1884148081334
99.90 percentile latency (ns) : 1902868349502
================================================
Test Parameters Used
samples_per_query : 1633
target_qps : 1
ttft_latency (ns): 100000000
tpot_latency (ns): 100000000
max_async_queries : 1
min_duration (ms): 600000
max_duration (ms): 0
min_query_count : 1
max_query_count : 0
qsl_rng_seed : 1780908523862526354
sample_index_rng_seed : 14771362308971278857
schedule_rng_seed : 18209322760996052031
accuracy_log_rng_seed : 0
accuracy_log_probability : 0
accuracy_log_sampling_target : 0
print_timestamps : 0
performance_issue_unique : 0
performance_issue_same : 0
performance_issue_same_index : 0
performance_sample_count : 1633
WARNING: sample_concatenate_permutation was set to true.
Generated samples per query might be different as the one in the setting.
Check the generated_samples_per_query line in the detailed log for the real
samples_per_query value
No warnings encountered during test.
1633 ERRORS encountered. See detailed log.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by rerunning sh ./reference_mlperf_accuracy.sh and inspect its detailed log, since the report only summarizes 1633 errors despite a VALID result. Compare the detailed errors with the PySUT, Offline, and PerformanceOnly settings shown here; done means identifying the cause and making the accuracy run complete without those errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, shell
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100