Lightning-AI/torchmetrics

View on GitHub

Machine learning metrics for distributed, scalable PyTorch applications.

Stars: 2.5k
Forks: 526
Open beginner issues: 2
Indexed issues: 106
Avg merge: 6d 11h
Merged PRs (30d): 5

Dominant language: Python
License: Apache-2.0
Last GitHub push: Sep 9, 2026
Latest indexed: Sep 19, 2026
Contributing guide: Contributing guide
Code of conduct: Code of conduct
Beginner labels: help wanted good first issue

Beginner friendly Everything

Open All

106 open issues indexed

Add an optional ultrafast-pycocotools backend for COCO mean average precision Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

Lightning-AI/torchmetrics#3499 · 1 comment · Opened Sep 10, 2026
AdjustedMutualInfoScore mishandles perfect-match limit cases Open

Difficulty 3/5 1-2 days Newbie friendliness 68/100

Lightning-AI/torchmetrics#3495 · Opened Sep 8, 2026
One-sided empty inputs are weighted inconsistently in intersection metrics Open

Difficulty 3/5 1-2 days Newbie friendliness 68/100

Lightning-AI/torchmetrics#3493 · Opened Sep 8, 2026
Group fairness metrics can relabel missing group IDs Open

Difficulty 3/5 1-2 days Newbie friendliness 45/100

Lightning-AI/torchmetrics#3489 · Opened Sep 8, 2026
`Metric.forward()` drops previously accumulated state when batch computation raises Open

Difficulty 4/5 3-5 days Newbie friendliness 42/100

Lightning-AI/torchmetrics#3487 · Opened Sep 8, 2026
`Metric.sync_context` leaves the metric synchronized when `compute()` raises Open

Difficulty 3/5 1-2 days Newbie friendliness 65/100

Lightning-AI/torchmetrics#3486 · 1 comment · Opened Sep 8, 2026
V-measure returns 1 for independent nontrivial clusterings Open

Difficulty 2/5 1-3 hours Newbie friendliness 76/100

Lightning-AI/torchmetrics#3484 · 1 comment · Opened Sep 8, 2026
Missing objects.inv in docs Open

documentation

Difficulty 3/5 1-2 days Newbie friendliness 48/100

Lightning-AI/torchmetrics#3475 · Opened Aug 21, 2026
ClasswiseWrapper silently drops values for invalid labels Open

Difficulty 2/5 1-3 hours Newbie friendliness 76/100

Lightning-AI/torchmetrics#3472 · 1 comment · Opened Aug 20, 2026
NMI/AMI return 0.0 instead of 1.0 for two single-cluster labelings Open

Difficulty 2/5 1-3 hours Newbie friendliness 75/100

Lightning-AI/torchmetrics#3469 · 1 comment · Opened Aug 20, 2026
davies_bouldin_score returns a wrong value for off-origin float64 data (float32 buffers) Open

Difficulty 2/5 1-3 hours Newbie friendliness 55/100

Lightning-AI/torchmetrics#3467 · Opened Aug 19, 2026
spearman_corrcoef and kendall_rank_corrcoef silently downcast float64 to float32 Open

Difficulty 3/5 1-2 days Newbie friendliness 75/100

Lightning-AI/torchmetrics#3465 · 2 comments · Opened Aug 18, 2026
Functional nominal metrics raise RuntimeError when category labels do not start at 0 Open

bug / fix help wanted

Difficulty 3/5 1-2 days Newbie friendliness 72/100

Lightning-AI/torchmetrics#3460 · 1 comment · Opened Aug 17, 2026
Import time: `scipy.signal`, `torchvision` and `matplotlib` are imported eagerly by `import torchmetrics` Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

Lightning-AI/torchmetrics#3457 · 2 comments · Opened Aug 15, 2026
Functional nominal metrics fail for non-zero-based category labels Open

Difficulty 3/5 1-2 days Newbie friendliness 68/100

Lightning-AI/torchmetrics#3446 · 1 comment · Opened Aug 9, 2026
Dependency resolving to the latest torchmetrics Open

bug / fix help wanted

Difficulty 3/5 1-2 days Newbie friendliness 50/100

Lightning-AI/torchmetrics#3439 · 1 comment · Opened Jul 30, 2026
functional/text/eed.py still carries the RWTH non-commercial licence (same class as #2665) Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

Lightning-AI/torchmetrics#3435 · 1 comment · Opened Jul 24, 2026
nan_stategy=disable dropping weights Open

bug / fix help wanted

Difficulty 3/5 1-2 days Newbie friendliness 35/100

Lightning-AI/torchmetrics#3434 · 1 comment · Opened Jul 23, 2026
Interpolation bug for macro-averaged multiclass precision-recall curves (`torchmetrics.functional.classification.precision_recall_curve._multiclass_precision_recall_curve_compute`) Open

bug / fix help wanted

Difficulty 2/5 1-3 hours Newbie friendliness 74/100

Lightning-AI/torchmetrics#3413 · 4 comments · Opened Jun 12, 2026
torchmetrics.retrieval.RetrievalPrecisionRecallCurve plots precision calls it False Positive Rate Open

bug / fix help wanted

Difficulty 3/5 1-2 days Newbie friendliness 55/100

Lightning-AI/torchmetrics#3403 · 3 comments · Opened Jun 2, 2026
mAP Metric for 3D Bounding Boxes Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 38/100

Lightning-AI/torchmetrics#3398 · 1 comment · Opened May 25, 2026
Slash-command workflows are non-functional (Slash Command Dispatch fails on every run) Open

bug / fix help wanted

Difficulty 4/5 3-5 days Newbie friendliness 38/100

Lightning-AI/torchmetrics#3393 · Opened May 21, 2026
`RetrievalMAP` and `RetrievalMRR` do not validate invalid top_k values Open

bug / fix help wanted

Difficulty 3/5 1-2 days Newbie friendliness 72/100

Lightning-AI/torchmetrics#3378 · Opened May 17, 2026
NTA-IoU and NTL-IoU metrics Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 48/100

Lightning-AI/torchmetrics#3366 · Opened Apr 30, 2026
PSNRB: `block_size` input validation is broken Open

bug / fix help wanted

Difficulty 1/5 1-3 hours Newbie friendliness 72/100

Lightning-AI/torchmetrics#3364 · 2 comments · Opened Apr 24, 2026
MCC Multilabel with average macro Open

enhancement

Difficulty 3/5 1-2 days Newbie friendliness 68/100

Lightning-AI/torchmetrics#3363 · 2 comments · Opened Apr 21, 2026
Bug Open

bug / fix help wanted

Difficulty 5/5 Over a week Newbie friendliness 15/100

Lightning-AI/torchmetrics#3362 · 1 comment · Opened Apr 19, 2026
BinaryMatthewsCorrCoef behaves inconsistently when denominator = 0 Open

bug / fix help wanted

Difficulty 4/5 3-5 days Newbie friendliness 45/100

Lightning-AI/torchmetrics#3355 · Opened Apr 4, 2026
Multiclass accuracy requires num_classes**2 memory Open

bug / fix help wanted

Difficulty 2/5 1-3 hours Newbie friendliness 74/100

Lightning-AI/torchmetrics#3343 · 1 comment · Opened Mar 21, 2026
`_sync_dist` hangs for `dist_reduce_fx=None` list states when ranks have different list lengths Open

bug / fix help wanted

Difficulty 5/5 Over a week Newbie friendliness 35/100

Lightning-AI/torchmetrics#3336 · 2 comments · Opened Mar 16, 2026
MetricCollection not working with MeanAveragePrecision Open

bug / fix help wanted

Difficulty 3/5 1-2 days Newbie friendliness 45/100

Lightning-AI/torchmetrics#3335 · Opened Mar 16, 2026
`spectral_angle_mapper` NaNs even with one zero pixel Open

bug / fix help wanted

Difficulty 3/5 1-2 days Newbie friendliness 48/100

Lightning-AI/torchmetrics#3322 · Opened Feb 7, 2026
Failed to import from `torchmetrics.image.lpip` Open

bug / fix help wanted

Difficulty 3/5 1-2 days Newbie friendliness 35/100

Lightning-AI/torchmetrics#3314 · 1 comment · Opened Jan 9, 2026
Can't scroll the left menu from the docs page Open

documentation

Difficulty 3/5 1-2 days Newbie friendliness 45/100

Lightning-AI/torchmetrics#3313 · 1 comment · Opened Dec 31, 2025
Add a new metric: Localization Recall Precision (LRP) Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

Lightning-AI/torchmetrics#3305 · 2 comments · Opened Nov 6, 2025
hausdorff_distance() throws IndexError for certain input shapes Open

bug / fix help wanted

Difficulty 3/5 1-2 days Newbie friendliness 52/100

Lightning-AI/torchmetrics#3300 · 4 comments · Opened Oct 29, 2025
MulticlassPrecisionRecallCurve with thresholds=[integer] does not use constant memory on MPS Open

bug / fix help wanted

Difficulty 3/5 1-2 days Newbie friendliness 55/100

Lightning-AI/torchmetrics#3299 · Opened Oct 27, 2025
Installing torchmetrics causes GIL error even without usage Open

bug / fix help wanted

Difficulty 4/5 3-5 days Newbie friendliness 35/100

Lightning-AI/torchmetrics#3296 · Opened Oct 18, 2025
Memory Leak in AveragePrecision Open

bug / fix help wanted

Difficulty 4/5 3-5 days Newbie friendliness 42/100

Lightning-AI/torchmetrics#3294 · 3 comments · Opened Oct 16, 2025
BinaryAccuracy supports weight parameter Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 30/100

Lightning-AI/torchmetrics#3274 · 1 comment · Opened Sep 23, 2025
LPIPS turn slow during training Open

bug / fix help wanted

Difficulty 3/5 1-2 days Newbie friendliness 52/100

Lightning-AI/torchmetrics#3267 · 2 comments · 2 reactions · Opened Sep 18, 2025
Intersection Over Union (IoU) should support Shape N x C_1 x C_2 x ... x C_n Open

bug / fix help wanted

Difficulty 3/5 1-2 days Newbie friendliness 72/100

Lightning-AI/torchmetrics#3262 · 1 comment · Opened Sep 13, 2025
Add Vendi Score and Memorization metrics for generative model evaluation Open

enhancement

Lightning-AI/torchmetrics#3233 · 1 comment · 1 reaction · 1 assignee · Opened Aug 26, 2025
MeanAveragePrecision for custom area definitions Open

enhancement

Difficulty 3/5 1-2 days Newbie friendliness 52/100

Lightning-AI/torchmetrics#3231 · 1 comment · Opened Aug 19, 2025
`retrieval_normalized_dcg` should compute per-query average when given 2D inputs (IR-standard behavior) Open

bug / fix help wanted

Lightning-AI/torchmetrics#3216 · 3 comments · 1 assignee · Opened Aug 10, 2025
`MeanAveragePrecision(sync_on_compute=True)` hangs when using `LightningModule` with `ray`'s `TorchTrainer` Open

bug / fix help wanted waiting on author

Difficulty 4/5 3-5 days Newbie friendliness 30/100

Lightning-AI/torchmetrics#3199 · 4 comments · 2 reactions · Opened Jul 31, 2025
Robust instance segmentation metrics Open

enhancement New metric

Lightning-AI/torchmetrics#3163 · 8 comments · 1 assignee · Opened Jul 3, 2025
Frechet Inception Distance with Dinov2 features Open

enhancement

Lightning-AI/torchmetrics#3110 · 8 comments · 1 assignee · Opened May 27, 2025
Upper Face Dynamic Deviation (FDD) Metrics for 3D Talking Heads Evaluation Open

enhancement

Lightning-AI/torchmetrics#3097 · 4 comments · 1 reaction · 1 assignee · Opened May 14, 2025
Provide BinaryAUROC support for Masked/Sparse Labels Open

enhancement

Lightning-AI/torchmetrics#3096 · 5 comments · 1 assignee · Opened May 14, 2025

Lightning-AI/torchmetrics

Get new issues in your inbox