tensorflow / tensorflow/tensorboard

Add UI support for min/max aggregation across runs in a trial

Open
#2,464 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

plugin:hparams type:feature
Dominant language
TypeScript
Stars
7.2k
Forks
1.7k
Avg merge
4d 22h
Merged PRs (30d)
1

Description

The hparams dashboard groups multiple runs with the same hyperparameters
(but different random seed) into a single “trial”. Currently, the
metrics shown in the frontend are always averaged across runs in a
trial. This is perhaps a reasonable default, but it’s somewhat
misleading in that shows metrics that may not be consistent with any
feasible run.

The hparams backend already supports smarter aggregation modes that
select a “best” run from a trial and show consistent metrics from that
run. The best run is chosen by a user-defined metric and aggregation
direction: e.g., ^highest accuracy” or “lowest xent”.

The only problem is that the UI has no controls to actually use this
functionality, so we should add that. :-)

See: API proto definition; backend implementation;
discussion on #1998.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the aggregation modes in tensorboard/plugins/hparams/api.proto and the backend implementation in tensorboard/plugins/hparams/list_session_groups.py, then trace the hparams frontend controls that consume them. Done means users can choose a trial aggregation metric and direction in the UI and see the selected consistent run metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
frontend
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.