mlcommons / mlcommons/inference
Define standard for representing and filtering heterogeneous systems on the official result table
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 650
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 6
Description
I would like to continue our discussion related to how we should display heterogeneous system results here. Given that there have already been two submitters with heterogeneous GPU configurations (e.g., systems with "GPU A + GPU B"), it is a good time to find a standard for how heterogeneous systems are shown and represented in the result table.
The goal of this is to make sure readers can:
- Still see apples-to-apples comparisons
- Make sure that no system is left out with the filtering logic in the results viewer (ex: when a user filters for "GPU A", a system listed as "GPU A + GPU B" is hidden. This prevents users from discovering all relevant results). The current table lists the heterogeneous system as a separate "accelerator type", which I believe is less than ideal.
- Does not clutter the default view of the results table.
The current format gets us 1) and 3), but loses the ability to do 2). Let's use this issue to discuss and define this standard. Possible approaches could be:
-
Show the heterogeneous results when one of the accelerator is picked for the filtering.
-
Allow submitters to tag submitter a "primary" accelerator designation, and the result will show up if that tagged accelerator is used as the filter.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the current official result table and results viewer filtering behavior described in the issue. Define a representation and filtering standard for heterogeneous accelerator systems that preserves apples-to-apples comparisons, includes relevant results when filtering, and avoids cluttering the default view; done means the standard is agreed and its affected behavior is specified.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-visualization
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100