mlcommons / mlcommons/inference

Define standard for representing and filtering heterogeneous systems on the official result table

Open
#2,379 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.6k
Forks
650
Avg merge
1d 22h
Merged PRs (30d)
6

Description

I would like to continue our discussion related to how we should display heterogeneous system results here. Given that there have already been two submitters with heterogeneous GPU configurations (e.g., systems with "GPU A + GPU B"), it is a good time to find a standard for how heterogeneous systems are shown and represented in the result table.

The goal of this is to make sure readers can:

  1. Still see apples-to-apples comparisons
  2. Make sure that no system is left out with the filtering logic in the results viewer (ex: when a user filters for "GPU A", a system listed as "GPU A + GPU B" is hidden. This prevents users from discovering all relevant results). The current table lists the heterogeneous system as a separate "accelerator type", which I believe is less than ideal.
  3. Does not clutter the default view of the results table.

The current format gets us 1) and 3), but loses the ability to do 2). Let's use this issue to discuss and define this standard. Possible approaches could be:

  • Show the heterogeneous results when one of the accelerator is picked for the filtering.

  • Allow submitters to tag submitter a "primary" accelerator designation, and the result will show up if that tagged accelerator is used as the filter.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the current official result table and results viewer filtering behavior described in the issue. Define a representation and filtering standard for heterogeneous accelerator systems that preserves apples-to-apples comparisons, includes relevant results when filtering, and avoids cluttering the default view; done means the standard is agreed and its affected behavior is specified.

Written by the indexing model from the issue text.

Assessment

Domain
data-visualization
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.