mlcommons / mlcommons/modelbench
Better observability tooling for big runs
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 134
- Forks
- 36
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 17
Description
The problem:
Currently, the output of Modelbench during a run is primarily progress bars showing the progress of test items against a SUT. This is moderately useful when running everything sequentially, but especially as we attempt to run these jobs in parallel, this becomes difficult or impossible to parse.
The solution(?):
Well, we're not sure yet. But, ideally the solution would let us do things including, but not limited to:
- being able to see the progress of test items against all SUTs, regardless of whether they're being run in sequence or in parallel and;
- being able to see the overall progress of the benchmark run and;
- being able to see how long the run has been running and;
- being able to see, maybe, very approximately, how long the run anticipates continuing to run and;
- being able to see where the run has encountered problems, what kinds of problems those are, and maybe if there's a way to resolve them
I've done some very rough sketching with a library called Rich to see if it'd be useful and came up with something like this (which was mostly working-ish), but I don't think anyone is married to this particular idea:
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing Modelbench's current progress-bar output during sequential and parallel runs, then inspect the existing Rich sketch referenced in the issue. Done means an agreed observability approach that shows benchmark and test-item progress, elapsed and approximate remaining time, and run problems with useful resolution context.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100