pytorch / pytorch/ignite

Make sure metric handlers are fired first

Open
#706 7 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
4.8k
Forks
726
Avg merge
5d 21h
Merged PRs (30d)
5

Description

Hi,

I noticed that since event handlers are invoked at the order of their insertion, there's a potential bug- if a metric handler is inserted after a TensorBoard handler, the metric value will not be ready at the time of writing.

While this has only minor effects (one epoch shift in the results), I think it can be easily solved by one of two approaches:

  1. separating handlers list to two: one for metrics, and the other handler types.
  2. More generic and scalable- allowing users to decide the order of execution by adding priority argument to add_event_handler (default would be that metric handlers are inserted with higher priority) and then sort the list when run() is invoked.

Let me know if you like any of the ideas, and I'd be happy to draft a PR

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing add_event_handler and run(), focusing on how handlers are stored and invoked. Decide how metric handlers should be ordered relative to TensorBoard handlers, then verify that metric values are ready before writing even when handlers were added in the opposite order.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.