pytorch / pytorch/ignite

Metrics general supplementary checking before `compute`

Open
#335 19 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement needs-discussion
Dominant language
Python
Stars
4.8k
Forks
726
Avg merge
5d 21h
Merged PRs (30d)
5

Description

As discussed here, the following assumption:

All metrics that has not been updated yet after last reset should raise NotComputableError when compute is called

holds for any metric.
Today we check metric specific variable _num_examples if it's equal zero and raise exception if it is the case. This behaviour can be replaced directly in Metric by counting the number of update calls between reset and before compute.
An example implementation is proposed by @zasdfgbnm :

class Metric:
    def  __init__(self, ...):
        ......

        def wrapped_reset():
            self._updated = False
            self.reset()
        self.reset = functools.wraps(self.reset)(wrapped_reset)

        def wrapped_update(output):
            self._updated = True
            self.update(output)
        self.update =  functools.wraps(self.update)(wrapped_update)

        def wrapped_compute():
            if not self._updated:
                raise NotComputableError('not updated before compute')
            return self.compute()
        self.compute =  functools.wraps(self.compute)(wrapped_compute)

Let's discuss here what we can do.

cc @zasdfgbnm

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the Metric implementation and the metric-specific _num_examples checks described in the issue, then review the linked pull request discussion for the proposed wrapping approach. Done means the pre-compute update check is handled centrally in Metric and the existing NotComputableError behavior is preserved for metrics that were not updated.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.