tensorflow / tensorflow/tensorboard

Tensorboard not displaying all the HParams events

Open
#2,942 19 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

core:backend core:frontend plugin:hparams stat:awaiting tensorflower type:support
Dominant language
TypeScript
Stars
7.2k
Forks
1.7k
Avg merge
4d 22h
Merged PRs (30d)
1

Description

This is my logs directory structure :
image

There are two Jupyter notebooks running in parallel which have the exact same code except for the prefix of the run-* directory. Both of them dump hparam_tuning metrics in the same directory. Both of them train the same model with same hyper-parameters and metrics, but on different data. My requirement is to view all these runs in same table of tensorboard.

EDIT NOTE : Training data in the code gets generated & processed ONLY once in one notebook for all the runs. I cannot read & process the data multiple times for different runs. Also, both these notebooks run on different GPUs, I want to run them in parallel which is why I cannot run them in one notebook since the runs are sequential.

Tensorboard reads only 9 of the 18 hparam runs that I have in my logs directory :
image

Although I am able to see the scalars which I am using to monitor the loss, which are in the same log directory. Moreover, the metrics for hyper-parameter tuning are also visible for all the runs under the "Scalar" tab, but not under HPARAMS tab.

image

EDIT NOTE : I have kept the identifier as another hyper-parameter to view hparam logs generated by both the Jupyter notebooks. It's just for filtering purposes, since there is no feature to filter them via trialId.

Is there any way I can merge the results of different session runs in one table?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the 18-run setup from the shared logs directory and compare what appears in the HParams and Scalar tabs. Trace the HParams tab's run-discovery and table population path; done means all 18 HParams runs from both notebooks appear together without losing the existing scalar results.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, typescript
Domain
data-visualization, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.