Project-MONAI / Project-MONAI/model-zoo

multi-gpu tensorboard handlers initialization

Open
#520 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
338
Forks
106
PR merge metrics
No merged PRs in 30d

Description

https://github.com/Project-MONAI/model-zoo/blob/cf5e0322ee25b178b6cf841f3bd81e0a8adf2b16/models/spleen_ct_segmentation/configs/multi_gpu_train.json#L18

the multi-gpu override essentially set the trainer handlers to $@train#handlers[:-2] for the worker nodes. but because of the @train#handlers reference, the config parser will still trigger handler constructor calls on all nodes.

for tensorboard handlers this will be an issue, as each constructor call will create a new event log file. as a result the multinode log will have unnecessary event logging files. https://github.com/Project-MONAI/MONAI/blob/e36982b87bf87fb9559fc4d124e132b67f177d23/monai/handlers/tensorboard_handlers.py#L52-L55

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with models/spleen_ct_segmentation/configs/multi_gpu_train.json at line 18 and trace how the @train#handlers reference is parsed for worker nodes. Read MONAI's tensorboard_handlers.py around lines 52-55 to understand why constructor calls create event logs. Done means multinode training no longer creates unnecessary TensorBoard event log files on worker nodes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.