Project-MONAI / Project-MONAI/model-zoo
multi-gpu tensorboard handlers initialization
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 338
- Forks
- 106
- PR merge metrics
- No merged PRs in 30d
Description
the multi-gpu override essentially set the trainer handlers to $@train#handlers[:-2] for the worker nodes. but because of the @train#handlers reference, the config parser will still trigger handler constructor calls on all nodes.
for tensorboard handlers this will be an issue, as each constructor call will create a new event log file. as a result the multinode log will have unnecessary event logging files. https://github.com/Project-MONAI/MONAI/blob/e36982b87bf87fb9559fc4d124e132b67f177d23/monai/handlers/tensorboard_handlers.py#L52-L55
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with models/spleen_ct_segmentation/configs/multi_gpu_train.json at line 18 and trace how the @train#handlers reference is parsed for worker nodes. Read MONAI's tensorboard_handlers.py around lines 52-55 to understand why constructor calls create event logs. Done means multinode training no longer creates unnecessary TensorBoard event log files on worker nodes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100