Lightning-AI / Lightning-AI/torchmetrics

MetricCollection not working with MeanAveragePrecision

Open
#3,335 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug / fix help wanted
Dominant language
Python
Stars
2.5k
Forks
526
Avg merge
6d 11h
Merged PRs (30d)
5

Description

## 🐛 Bug

MetricCollection does not work with MeanAveragePrecision.

### To Reproduce

Sample code and steps to reproduce the behavior with expected result...

```python
from torch import tensor
import torch
from torchmetrics.detection import MeanAveragePrecision
from torchmetrics.collections import MetricCollection

mask_pred = [
[0, 0, 0, 0, 0],
[0, 0, 1, 1, 0],
[0, 0, 1, 1, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
]
mask_tgt = [
[0, 0, 0, 0, 0],
[0, 0, 1, 0, 0],
[0, 0, 1, 1, 0],
[0, 0, 1, 0, 0],
[0, 0, 0, 0, 0],
]
preds = [
dict(
masks=tensor([mask_pred, mask_pred], dtype=torch.bool),
boxes=tensor([[258.0, 41.0, 606.0, 285.0], [258.0, 41.0, 606.0, 285.0]]),
scores=tensor([0.536, 0.4]),
labels=tensor([0, 0]),
)
]
target = [
dict(
masks=tensor([mask_tgt, mask_tgt], dtype=torch.bool),
boxes=tensor([[214.0, 41.0, 562.0, 285.0], [214.0, 41.0, 562.0, 285.0]]),
labels=tensor([0, 1]),
)
]

metric_map = MeanAveragePrecision(iou_type=["segm", "bbox"], class_metrics=True)
metric_map_50 = MeanAveragePrecision(
iou_type=["segm", "bbox"], iou_thresholds=[50], class_metrics=True
)
metric = MetricCollection({"iou_all": metric_map, "iou_50": metric_map_50})
metric.update(preds, target)

res = metric.compute()
```

- TorchMetrics version: 1.8.2

### Additional context

Error message
```
metric.update(preds, target)
File "/home/user/miniconda3/envs/cv-classification2/lib/python3.10/site-packages/torchmetrics/collections.py", line 263, in update
self._merge_compute_groups()
File "/home/user/miniconda3/envs/cv-classification2/lib/python3.10/site-packages/torchmetrics/collections.py", line 286, in _merge_compute_groups
if self._equal_metric_states(metric1, metric2):
File "/home/user/miniconda3/envs/cv-classification2/lib/python3.10/site-packages/torchmetrics/collections.py", line 332, in _equal_metric_states
and not (all(s1.shape == s2.shape and allclose(s1, s2) for s1, s2 in zip(state1, state2)))
File "/home/user/miniconda3/envs/cv-classification2/lib/python3.10/site-packages/torchmetrics/collections.py", line 332, in
and not (all(s1.shape == s2.shape and allclose(s1, s2) for s1, s2 in zip(state1, state2)))
AttributeError: 'tuple' object has no attribute 'shape'
```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in torchmetrics/collections.py at MetricCollection.update, then inspect _merge_compute_groups and _equal_metric_states, where the traceback shows the tuple handling failure. Run the provided MeanAveragePrecision reproduction with TorchMetrics 1.8.2; done means metric.update and metric.compute complete without the tuple shape error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.