Lightning-AI / Lightning-AI/torchmetrics
MetricCollection not working with MeanAveragePrecision
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 526
- Avg merge
- 6d 11h
- Merged PRs (30d)
- 5
Description
## 🐛 Bug
MetricCollection does not work with MeanAveragePrecision.
### To Reproduce
Sample code and steps to reproduce the behavior with expected result...
```python
from torch import tensor
import torch
from torchmetrics.detection import MeanAveragePrecision
from torchmetrics.collections import MetricCollection
mask_pred = [
[0, 0, 0, 0, 0],
[0, 0, 1, 1, 0],
[0, 0, 1, 1, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
]
mask_tgt = [
[0, 0, 0, 0, 0],
[0, 0, 1, 0, 0],
[0, 0, 1, 1, 0],
[0, 0, 1, 0, 0],
[0, 0, 0, 0, 0],
]
preds = [
dict(
masks=tensor([mask_pred, mask_pred], dtype=torch.bool),
boxes=tensor([[258.0, 41.0, 606.0, 285.0], [258.0, 41.0, 606.0, 285.0]]),
scores=tensor([0.536, 0.4]),
labels=tensor([0, 0]),
)
]
target = [
dict(
masks=tensor([mask_tgt, mask_tgt], dtype=torch.bool),
boxes=tensor([[214.0, 41.0, 562.0, 285.0], [214.0, 41.0, 562.0, 285.0]]),
labels=tensor([0, 1]),
)
]
metric_map = MeanAveragePrecision(iou_type=["segm", "bbox"], class_metrics=True)
metric_map_50 = MeanAveragePrecision(
iou_type=["segm", "bbox"], iou_thresholds=[50], class_metrics=True
)
metric = MetricCollection({"iou_all": metric_map, "iou_50": metric_map_50})
metric.update(preds, target)
res = metric.compute()
```
- TorchMetrics version: 1.8.2
### Additional context
Error message
```
metric.update(preds, target)
File "/home/user/miniconda3/envs/cv-classification2/lib/python3.10/site-packages/torchmetrics/collections.py", line 263, in update
self._merge_compute_groups()
File "/home/user/miniconda3/envs/cv-classification2/lib/python3.10/site-packages/torchmetrics/collections.py", line 286, in _merge_compute_groups
if self._equal_metric_states(metric1, metric2):
File "/home/user/miniconda3/envs/cv-classification2/lib/python3.10/site-packages/torchmetrics/collections.py", line 332, in _equal_metric_states
and not (all(s1.shape == s2.shape and allclose(s1, s2) for s1, s2 in zip(state1, state2)))
File "/home/user/miniconda3/envs/cv-classification2/lib/python3.10/site-packages/torchmetrics/collections.py", line 332, in
and not (all(s1.shape == s2.shape and allclose(s1, s2) for s1, s2 in zip(state1, state2)))
AttributeError: 'tuple' object has no attribute 'shape'
```
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in torchmetrics/collections.py at MetricCollection.update, then inspect _merge_compute_groups and _equal_metric_states, where the traceback shows the tuple handling failure. Run the provided MeanAveragePrecision reproduction with TorchMetrics 1.8.2; done means metric.update and metric.compute complete without the tuple shape error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100