mlcommons / mlcommons/modelbench

Better observability tooling for big runs

Open
#343 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
134
Forks
36
Avg merge
1d 11h
Merged PRs (30d)
17

Description

The problem:

Currently, the output of Modelbench during a run is primarily progress bars showing the progress of test items against a SUT. This is moderately useful when running everything sequentially, but especially as we attempt to run these jobs in parallel, this becomes difficult or impossible to parse.

The solution(?):

Well, we're not sure yet. But, ideally the solution would let us do things including, but not limited to:

  • being able to see the progress of test items against all SUTs, regardless of whether they're being run in sequence or in parallel and;
  • being able to see the overall progress of the benchmark run and;
  • being able to see how long the run has been running and;
  • being able to see, maybe, very approximately, how long the run anticipates continuing to run and;
  • being able to see where the run has encountered problems, what kinds of problems those are, and maybe if there's a way to resolve them

I've done some very rough sketching with a library called Rich to see if it'd be useful and came up with something like this (which was mostly working-ish), but I don't think anyone is married to this particular idea:

Image

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing Modelbench's current progress-bar output during sequential and parallel runs, then inspect the existing Rich sketch referenced in the issue. Done means an agreed observability approach that shows benchmark and test-item progress, elapsed and approximate remaining time, and run problems with useful resolution context.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.