tensorflow / tensorflow/tensorboard
Add UI support for min/max aggregation across runs in a trial
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 7.2k
- Forks
- 1.7k
- Avg merge
- 4d 22h
- Merged PRs (30d)
- 1
Description
The hparams dashboard groups multiple runs with the same hyperparameters
(but different random seed) into a single “trial”. Currently, the
metrics shown in the frontend are always averaged across runs in a
trial. This is perhaps a reasonable default, but it’s somewhat
misleading in that shows metrics that may not be consistent with any
feasible run.
The hparams backend already supports smarter aggregation modes that
select a “best” run from a trial and show consistent metrics from that
run. The best run is chosen by a user-defined metric and aggregation
direction: e.g., ^highest accuracy” or “lowest xent”.
The only problem is that the UI has no controls to actually use this
functionality, so we should add that. :-)
See: API proto definition; backend implementation;
discussion on #1998.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the aggregation modes in tensorboard/plugins/hparams/api.proto and the backend implementation in tensorboard/plugins/hparams/list_session_groups.py, then trace the hparams frontend controls that consume them. Done means users can choose a trial aggregation metric and direction in the UI and see the selected consistent run metrics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- frontend
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100