Lightning-AI / Lightning-AI/torchmetrics
Make `ignore_index` work when all batch elements are to be ignored
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 526
- Avg merge
- 6d 11h
- Merged PRs (30d)
- 5
Description
## 🚀 Feature
The `ignore_index` argument in e.g. the `AUROC` metric allows one to specify a label that will be ignored. This works great when some batch elements are to be ignored. When calling the metric, and providing a tensor as input where all entries are the `ignore_index`, we get an `IndexError`.
```
self.aucs[f"val_label_{i}"](label_logits[:, i].squeeze(-1), labels_target[:, i])
File "lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "lib/python3.10/site-packages/torchmetrics/metric.py", line 312, in forward
self._forward_cache = self._forward_reduce_state_update(*args, **kwargs)
File "lib/python3.10/site-packages/torchmetrics/metric.py", line 382, in _forward_reduce_state_update
batch_val = self.compute()
File "/lib/python3.10/site-packages/torchmetrics/metric.py", line 633, in wrapped_func
value = _squeeze_if_scalar(compute(*args, **kwargs))
File "lib/python3.10/site-packages/torchmetrics/classification/auroc.py", line 124, in compute
return _binary_auroc_compute(state, self.thresholds, self.max_fpr)
File "lib/python3.10/site-packages/torchmetrics/functional/classification/auroc.py", line 89, in _binary_auroc_compute
fpr, tpr, _ = _binary_roc_compute(state, thresholds, pos_label)
File "lib/python3.10/site-packages/torchmetrics/functional/classification/roc.py", line 54, in _binary_roc_compute
fps, tps, thres = _binary_clf_curve(preds=state[0], target=state[1], pos_label=pos_label)
File "lib/python3.10/site-packages/torchmetrics/functional/classification/precision_recall_curve.py", line 72, in _binary_clf_curve
tps = _cumsum(target * weight, dim=0)[threshold_idxs]
```
### Motivation
Having batches without labels may sound counterintuitive at first, but in multitask problems this can happen quite easily, when a metric only tracks a given subtask and batches are random.
### Pitch
It would be helpful if this just worked (and maybe print a warning) - maybe return 0 or nan?
### Alternatives
Right now, this needs to be handled manually like
```python
if (target == -100).all():
pass
else:
self.auc(logits, target)
```
or, when calling `compute` after some `update` steps
```python
if all([len(x)==0 for x in self.auc.metric_state['preds']]):
pass
else:
self.auc.compute()
```
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the AUROC and ignore_index handling referenced in classification/auroc.py, then follow the traceback through functional/classification/roc.py and precision_recall_curve.py. Reproduce the all-ignored target case and establish the expected result for that input; done means it no longer raises IndexError and has regression coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100