Make sure metric handlers are fired first
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.8k
- Forks
- 726
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 5
Description
Hi,
I noticed that since event handlers are invoked at the order of their insertion, there's a potential bug- if a metric handler is inserted after a TensorBoard handler, the metric value will not be ready at the time of writing.
While this has only minor effects (one epoch shift in the results), I think it can be easily solved by one of two approaches:
- separating handlers list to two: one for metrics, and the other handler types.
- More generic and scalable- allowing users to decide the order of execution by adding priority argument to
add_event_handler(default would be that metric handlers are inserted with higher priority) and then sort the list whenrun()is invoked.
Let me know if you like any of the ideas, and I'd be happy to draft a PR
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing add_event_handler and run(), focusing on how handlers are stored and invoked. Decide how metric handlers should be ordered relative to TensorBoard handlers, then verify that metric values are ready before writing even when handlers were added in the opposite order.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100