Lightning-AI / Lightning-AI/torchmetrics

Make `ignore_index` work when all batch elements are to be ignored

Open
#2,685 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
2.5k
Forks
526
Avg merge
6d 11h
Merged PRs (30d)
5

Description

## 🚀 Feature

The `ignore_index` argument in e.g. the `AUROC` metric allows one to specify a label that will be ignored. This works great when some batch elements are to be ignored. When calling the metric, and providing a tensor as input where all entries are the `ignore_index`, we get an `IndexError`.

```
self.aucs[f"val_label_{i}"](label_logits[:, i].squeeze(-1), labels_target[:, i])
File "lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "lib/python3.10/site-packages/torchmetrics/metric.py", line 312, in forward
self._forward_cache = self._forward_reduce_state_update(*args, **kwargs)
File "lib/python3.10/site-packages/torchmetrics/metric.py", line 382, in _forward_reduce_state_update
batch_val = self.compute()
File "/lib/python3.10/site-packages/torchmetrics/metric.py", line 633, in wrapped_func
value = _squeeze_if_scalar(compute(*args, **kwargs))
File "lib/python3.10/site-packages/torchmetrics/classification/auroc.py", line 124, in compute
return _binary_auroc_compute(state, self.thresholds, self.max_fpr)
File "lib/python3.10/site-packages/torchmetrics/functional/classification/auroc.py", line 89, in _binary_auroc_compute
fpr, tpr, _ = _binary_roc_compute(state, thresholds, pos_label)
File "lib/python3.10/site-packages/torchmetrics/functional/classification/roc.py", line 54, in _binary_roc_compute
fps, tps, thres = _binary_clf_curve(preds=state[0], target=state[1], pos_label=pos_label)
File "lib/python3.10/site-packages/torchmetrics/functional/classification/precision_recall_curve.py", line 72, in _binary_clf_curve
tps = _cumsum(target * weight, dim=0)[threshold_idxs]
```

### Motivation

Having batches without labels may sound counterintuitive at first, but in multitask problems this can happen quite easily, when a metric only tracks a given subtask and batches are random.

### Pitch

It would be helpful if this just worked (and maybe print a warning) - maybe return 0 or nan?

### Alternatives

Right now, this needs to be handled manually like
```python
if (target == -100).all():
pass
else:
self.auc(logits, target)
```

or, when calling `compute` after some `update` steps
```python
if all([len(x)==0 for x in self.auc.metric_state['preds']]):
pass
else:
self.auc.compute()
```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the AUROC and ignore_index handling referenced in classification/auroc.py, then follow the traceback through functional/classification/roc.py and precision_recall_curve.py. Reproduce the all-ignored target case and establish the expected result for that input; done means it no longer raises IndexError and has regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.