Lightning-AI / Lightning-AI/torchmetrics

Multi Label Precision numerical stability issues with MPS

Open
#2,955 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug / fix duplicate help wanted v1.6.x
Dominant language
Python
Stars
2.5k
Forks
526
Avg merge
6d 11h
Merged PRs (30d)
5

Description

## 🐛 Bug

Currently trying to track metrics of a rather complicated ECG multi-label classifier (20 labels) and noticed that training with MPS breaks the MultiLabelPrecision in an odd way, so that the returned values are clearly outside of any possible scope e.g. -1.49272e+25. First I thought it is a numerical issue on the logits or some weird input shape issue I am not handling properly, but after doing some further digging I noticed that the inconsistency happens with device MPS specifically and can be reproduced. I couldn't get to the bottom of what leads to this as it is inconsistent, but most likely numerical.

Would love some guidance on this!

### To Reproduce

Below code snippet with manually calculated expected outputs. I hope my math isn't wrong, but nonetheless, if I set device to MPS and run the code a few times (may work on some runs) I will eventually run into some very odd numerical issues on test 3 and 4:

Code sample

```python
import torch
import torchmetrics
import logging

logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)

def test_label_scenarios(device='cpu'):
num_labels = 20

# Test scenarios specifically for 20-label case
test_cases = [
{
"name": "All labels correct",
"preds": torch.full((100, 20), 0.7),
"target": torch.ones((100, 20)),
"expected": 1.0,
},
{
"name": "All labels wrong",
"preds": torch.full((100, 20), 0.7),
"target": torch.zeros((100, 20)),
"expected": 0.0,
},
{
"name": "Exactly 5 labels correct per sample",
"preds": torch.cat(
[
torch.full((100, 5), 0.9), # 5 correct predictions
torch.full((100, 15), 0.1), # 15 never predicted positive
],
dim=1,
),
"target": torch.cat(
[
torch.ones((100, 5)),
torch.zeros((100, 15))
],
dim=1
),
# For macro, 5 labels have precision=1, 15 labels have precision=0 => avg=0.25
"expected": 0.25,
},
{
"name": "5 correct + 5 false positives",
"preds": torch.cat(
[
torch.full((100, 5), 0.9), # 5 always correct
torch.full((100, 5), 0.9), # 5 always false
torch.full((100, 10), 0.1),
],
dim=1,
),
"target": torch.cat(
[
torch.ones((100, 5)), # True positives for first 5 labels
torch.zeros((100, 15)) # False (and predicted + negative)
],
dim=1
),
# For macro, first 5 => precision=1, next 5 => precision=0, last 10 => 0 => avg=0.25
"expected": 0.25,
},
{
"name": "All False Predictions",
"preds": torch.ones((1000, 20)), # All predictions=1
"target": torch.zeros((1000, 20)), # All labels=0
"expected": 0.0,
"checks": ["no_nan", "exact_zero"],
},
{
"name": "Numerical instability test",
"preds": torch.randn(5000, 20).sigmoid(),
"target": torch.randint(0, 2, (5000, 20)),
"expected": None,
},
]

for case in test_cases:
print(f"\n=== {case['name']} ===")

preds = case["preds"].to(device)
target = case["target"].to(device).long()

precision = torchmetrics.Precision(
task="multilabel",
num_labels=num_labels,
average="macro",
threshold=0.5
).to(device)

try:
# Simulate batch updates
for _ in range(10):
precision.update(preds, target)

result = precision.compute()
logger.debug(f"Computed precision: {result}")

if case["expected"] is not None:
assert torch.isclose(
result, torch.tensor(case["expected"], device=device),
atol=1e-7
), f"Expected {case['expected']}, got {result:.4f}"

# Check for numerical stability
assert not torch.isnan(result).any(), "NaN detected in precision"
assert result.abs().max() < 1e6, f"Precision value exploded: {result}"

print(f"✓ PASSED - Final Precision: {result:.4f}")
except Exception as e:
print(f"✗ FAILED: {str(e)}")
finally:
precision.reset()

if __name__ == "__main__":
test_label_scenarios('mps')

```

### Expected behavior

Test 3 and 4 should return this using MPS:

=== Exactly 5 labels correct per sample ===
DEBUG:__main__:Computed precision: 0.25
✓ PASSED - Final Precision: 0.2500

=== 5 correct + 5 false positives ===
DEBUG:__main__:Computed precision: 0.25
✓ PASSED - Final Precision: 0.2500

instead of something like:

=== Exactly 5 labels correct per sample ===
DEBUG:__main__:Computed precision: 16985032228864.0
✗ FAILED: Expected 0.25, got 16985032228864.0000

=== 5 correct + 5 false positives ===
DEBUG:__main__:Computed precision: -1.4927448875054142e+25
✗ FAILED: Expected 0.25, got -14927448875054142008066048.0000

### Environment

- TorchMetrics version (if build from source, add commit SHA): 1.6.1
- Python & PyTorch Version (e.g., 1.0): 3.12.5
- Any other relevant information such as OS (e.g., Linux): MacOS Sonoma 14.7.4, M3 Max, but should be reproducible on all Apple Silicon)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the provided Python reproduction for MultiLabelPrecision on the MPS device, comparing its repeated-update behavior with CPU. Check the metric's multilabel precision computation and add a regression test covering the five-label scenarios and numerical-stability case. Done means MPS returns the expected 0.25 values without NaN or extreme results.

Written by the indexing model from the issue text.

Assessment

Tech stack
macos, python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.