Metrics general supplementary checking before `compute`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.8k
- Forks
- 726
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 5
Description
As discussed here, the following assumption:
All metrics that has not been updated yet after last reset should raise NotComputableError when
computeis called
holds for any metric.
Today we check metric specific variable _num_examples if it's equal zero and raise exception if it is the case. This behaviour can be replaced directly in Metric by counting the number of update calls between reset and before compute.
An example implementation is proposed by @zasdfgbnm :
class Metric:
def __init__(self, ...):
......
def wrapped_reset():
self._updated = False
self.reset()
self.reset = functools.wraps(self.reset)(wrapped_reset)
def wrapped_update(output):
self._updated = True
self.update(output)
self.update = functools.wraps(self.update)(wrapped_update)
def wrapped_compute():
if not self._updated:
raise NotComputableError('not updated before compute')
return self.compute()
self.compute = functools.wraps(self.compute)(wrapped_compute)
Let's discuss here what we can do.
cc @zasdfgbnm
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the Metric implementation and the metric-specific _num_examples checks described in the issue, then review the linked pull request discussion for the proposed wrapping approach. Done means the pre-compute update check is handled centrally in Metric and the existing NotComputableError behavior is preserved for metrics that were not updated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100