mlcommons / mlcommons/logging

Results summarizer fails when results folder is empty

Open
#236 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
43
Forks
60
Avg merge
19h 56m
Merged PRs (30d)
3

Description

This error occurs:

Traceback (most recent call last):
File "/opt/python3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/python3.9/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/mlperf_logging/result_summarizer/__main__.py", line 3, in <module>
result_summarizer.main()
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/mlperf_logging/result_summarizer/result_summarizer.py", line 626, in main
_update_summaries(args.folder)
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/mlperf_logging/result_summarizer/result_summarizer.py", line 597, in _update_summaries
strong_scaling_summary, weak_scaling_summary = summarize_results(
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/mlperf_logging/result_summarizer/result_summarizer.py", line 517, in summarize_results
strong_scaling_scores = _compute_strong_scaling_scores(
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/mlperf_logging/result_summarizer/result_summarizer.py", line 330, in _compute_strong_scaling_scores
benchmark_scores[benchmark] = _compute_olympic_average(
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/mlperf_logging/result_summarizer/result_summarizer.py", line 185, in _compute_olympic_average
return sum_of_scores * 1.0 / len(countable_scores)
ZeroDivisionError: float division by zero

A suggested solution would be to add a previous check in package checker to avoid this "ugly" error

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with mlperf_logging/result_summarizer/result_summarizer.py, especially _compute_olympic_average, _compute_strong_scaling_scores, and _update_summaries; run the result summarizer against an empty results folder to reproduce the traceback. Done means the empty-folder case no longer raises the reported ZeroDivisionError and the existing summarization behavior remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.