pytorch / pytorch/ignite

Skip metric computation if output is None

Open
#1,065 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement module: metrics
Dominant language
Python
Stars
4.8k
Forks
726
Avg merge
5d 21h
Merged PRs (30d)
5

Description

🚀 Feature

Considering the context of https://github.com/pytorch/ignite/pull/1024
and another point about using tensor.item() on each iteration with TPU which slows down the computation, we need to have a way to skip using state.output if set to None.

In training on TPU

def training_step(e, b):
    # ....
    
    # instead of returning each iteration loss value
    # return loss.item()
    # we would like to do it 
    return loss.item() if (e.state.iteration - 1) % 20 == 0 else None 

however, if we have a running metric attached to the output it will fail if encounter None.

So, idea is to check engine.state.output here and return before doing any metric's update:
https://github.com/pytorch/ignite/blob/e3fc04e147db4c97a2ceb70597366d562e15c96f/ignite/metrics/metric.py#L123-L124

@sdesrozis @erip any thoughts ?

This isssue is related to https://github.com/pytorch/ignite/issues/996 but the difference is that we still want to execute all other handlers on iteration even if output is None.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in ignite/metrics/metric.py at the state.output handling referenced by the issue, and trace how metric updates are invoked during an iteration. Verify the behavior with an output of None: metric updates should be skipped while other iteration handlers still run; check the surrounding metric tests for the appropriate coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.