Lightning-AI / Lightning-AI/torchmetrics
Allow arbitrary types for metric states
@SkafteNicki is already working on this.
Since Apr 27, 2022.
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 526
- Avg merge
- 6d 11h
- Merged PRs (30d)
- 5
Description
🚀 Feature
Metric states seem to be limited to torch.Tensor or List[torch.Tensor].
In my usecase i want to store a dictionary as state. My dataset comprises of samples who can be assigned to different documents. In order to calculate macro metrics (calculate metrics per document and average) I want to store my metric states (e.g. true positives, false positives, etc.) as a dictionary. Here is some pseudocode:
class MyMetric(Metric):
def __init__(self, dist_sync_on_step: bool = False):
super().__init__(dist_sync_on_step=dist_sync_on_step)
self.add_state("statistics", default=defaultdict(lambda: defaultdict(float)), dist_reduce_fx=None)
def update(
self,
predictions: torch.Tensor,
targets: torch.Tensor,
document_ids: List
):
predictions = predictions.bool()
targets = targets.bool()
tps = predictions * targets
tns = predictions.logical_not() * targets.logical_not()
fps = predictions * targets.logical_not()
fns = predictions.logical_not() * targets
for id_, tp, tn, fp, fn in zip(document_ids, tps, tns, fps, fns):
self.statistics[id_]['tp'] += tp.float().item()
self.statistics[id_]['tn'] += tn.float().item()
self.statistics[id_]['fp'] += fp.float().item()
self.statistics[id_]['fn'] += fn.float().item()
def compute(self):
...
Unfortunately the above code is not allowed. Each metric state has to be a torch.Tensor or a List[torch.Tensor].
That means normal float values or numpy arrays cannot be used as metrics either. Is there a particular reason for that?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.