Lightning-AI / Lightning-AI/torchmetrics

Provide BinaryAUROC support for Masked/Sparse Labels

Open
#3,096 5 comments 0 reactions 1 assignee View on GitHub

@robertreaney is already working on this.

Since Jul 10, 2025.

enhancement
Dominant language
Python
Stars
2.5k
Forks
526
Avg merge
6d 11h
Merged PRs (30d)
5

Description

## 🚀 Feature

Currently the `BinaryAUROC` metric update step expects a square matrix at each iteration. In a multi-output training regiment involving masked outputs, the current AUROC metrics class fails to offer support.

This feature proposes to provide an additional metric, `MaskedBinaryAUROC`, to support calculating label-specific and aggregate AUROC when each batch step leverages sparse/nested/masked labels.

### Motivation

I had to create a custom class for this functionality for a professional project, and I'd like to contribute it. Also, my company would like to become involved in the open source community by promoting code when possible.

### Pitch

Create a `MaskedBinaryAUROC` metric with similar functionality to `AUROC`, but it will also take a mask at each iteration and only consider unmasked values in the final calculations.

The implementation is straightforward:

0. preds/targets/mask states with List defaults
1. `.update(preds, targets, mask)` by appending to the list
2. `.compute()` will iterate through each column to calculate a per-label value by leveraging `torch.metrics.functional.binary_auroc` after applying the mask
3. return mean of all column-wise auroc values

### Alternatives

- `SparseBinaryAUROC`: support a sparse tensor representation of the output labels instead of a dense output and a mask.
- Extend functionality of `BinaryAUROC` instead of making a separate class

### Additional context

There are additional related extensions of the `AUROC`, `MulticlassAUROC`, and `MultilabelAUROC` classes.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.