Project-MONAI / Project-MONAI/MONAI
EmbeddingCollapseMetric: per-class rank scores never reach `aggregate`, so asymmetric collapse is invisible in the summary score
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.7k
- Forks
- 1.6k
- Avg merge
- 5d 1h
- Merged PRs (30d)
- 20
Description
Describe the bug
In compute_embedding_collapse (monai/metrics/embedding_collapse.py), the aggregate score is built from a fixed key set:
primary = {"centroid_similarity", "effective_rank_score", "domain_shift", "separation"}
available = [v.to(device=emb.device) for k, v in scores.items() if k in primary and v is not None]
_per_class_rank emits its scores under per_class_rank_<cls>, so no per-class score is ever a member of primary, and none is ever included in available. The indicator is computed, returned in the dict, and then dropped before every reduction.
That is the one indicator whose documented purpose is catching collapse the global indicators structurally cannot see:
Detects asymmetric collapse: one class may use 400 dimensions while another collapses to 6, which global SVD would average away.
So under reduction="max" — documented as "the worst-case score (recommended for safety-critical use)" — the worst case is precisely the number that gets discarded. A caller who follows the docstring and monitors aggregate gets no signal from it at all.
To Reproduce
A healthy majority class, and a rare class whose encoder output has collapsed to a single point:
import torch
from monai.metrics.embedding_collapse import compute_embedding_collapse
torch.manual_seed(17)
n_major, n_minor, d = 200, 8, 64
major = torch.randn(n_major, d)
major[:, 0] -= 4.0 # centroid ~ -e1
minor = torch.zeros(n_minor, d)
minor[:, 0] = 8.0 # 8 identical rows, centroid = +e1
emb = torch.cat([major, minor])
lbl = torch.tensor([0] * n_major + [1] * n_minor, dtype=torch.long)
scores = compute_embedding_collapse(emb, lbl, reduction="max")
for k, v in scores.items():
print(f"{k:24s} {float(v) if v is not None else None}")
Measured output:
centroid_similarity 0.0057
per_class_rank_0 0.0485
per_class_rank_1 1.0000
separation 0.2658
effective_rank_score 0.0602
domain_shift None
aggregate 0.2658
Class 1 has exactly zero within-class variance, so per_class_rank_1 is 1.0 — the module's own maximum, via the sv_sum == 0 branch in _effective_rank_score. Yet aggregate remains 0.2658.
The global indicators are not misbehaving here. 200 of 208 samples are full-rank, so the global spectrum is barely perturbed (effective_rank_score == 0.0602); the centroids are anti-parallel, so centroid_similarity sits at the bottom of its range (0.0057). Every global view of this embedding space really is healthy. Only the per-class view sees that one class is dead — and that view never reaches the summary.
This is independent of #8996: every effective-rank call in this example has N > D (or exits via the zero-variance branch), so the 1/N floor does not apply.
Expected behavior
Per-class rank should participate in the reduction. Two ways to do it, and I would like a steer on which you guys prefer:
- Add each
per_class_rank_<cls>toavailableindividually. Correct undermax. Undermeanit re-weights the average by class count — C per-class terms against 4 global ones — so the meaning of the mean drifts as C grows. - Reduce per-class scores to one representative value first (
maxover classes, i.e. "the worst class"), then add that single value toavailable. Identical to option 1 undermax; keeps one vote per indicator undermean, regardless of C.
I have implemented option 2 in a PR, since it leaves reduction="mean" interpretable, but option 1 is a small change if you would rather every class count separately.
One note on the existing tests
test_healthy_embeddings_score_low builds each class from identical rows (+e1 x 10, -e1 x 10), so under either option both classes score per_class_rank == 1.0 and that fixture's aggregate becomes 1.0. The test only asserts on centroid_similarity, so it still passes — but the fixture is a genuinely zero-within-class-variance embedding space, so the new score is correct and the name is what is misleading. Happy to either give it within-class variance or rename it in the same PR, whichever you prefer.
Environment
dev branch (MONAI 1.6.0+).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in monai/metrics/embedding_collapse.py at compute_embedding_collapse and _per_class_rank, then run the reproduction and inspect test_healthy_embeddings_score_low. Done means per-class rank contributes to aggregate under the chosen reduction semantics, with tests covering the affected summary behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100