Lightning-AI / Lightning-AI/torchmetrics

add optional ignore_index in all metrics

Open
#2,273 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
2.5k
Forks
526
Avg merge
6d 11h
Merged PRs (30d)
5

Description

## 🚀 Feature
Could you please consider incorporating a systematically optional ignore_index argument in all metrics?

For instance, in classification tasks, we currently have the ability to use ignore_index as follows:

```python
torchmetrics.classification.MulticlassJaccardIndex(num_classes=15, ignore_index=255)
```
However, in the case of regression, there is currently no mechanism to utilize ignore_index when computing metrics such as MSE or MAE.
### Motivation

There is a plethora of datasets for regression tasks (but not limited to regression) that contain pixel values representing missing data. It is crucial to have the capability to ignore these values when computing metric statistics.

### Pitch

I suggest that all metrics be updated to implement an optional ignore_index argument.

### Alternatives

Is it feasible to incorporate the ignore_index functionality in advanced metric settings? For example, as shown in https://lightning.ai/docs/torchmetrics/stable/pages/overview.html#metric-kwargs

### Additional context

For instance, in the context of depth maps, ground truth data may contain missing values replaced with an arbitrary ignore_index, and the training loss is already designed to ignore them. This feature would provide consistency and flexibility across various metrics.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the existing ignore_index behavior in torchmetrics.classification.MulticlassJaccardIndex and the advanced metric settings described in the linked metric kwargs documentation. Define which metrics and regression cases are in scope, then verify that the optional argument consistently excludes the selected values while preserving existing metric behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.