NVIDIA / NVIDIA/physicsnemo

🐛[BUG]: Variance in metrics.general.ensemble_metrics ignores the dim argument

Open
#2,003 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
3.3k
Forks
787
Avg merge
2d 21h
Merged PRs (30d)
27

Description

Version

main at ff5d19d08123de47ca446caed1d70a225d540184

On which installation method(s) does this occur?

Source

Describe the issue

physicsnemo.metrics.general.ensemble_metrics.Variance.__call__ accepts a dim argument but two lines inside it still assume the ensemble dimension is the leading one.

  1. The sample count is taken from inputs.shape[0] instead of inputs.shape[dim]. Mean.__call__ a few lines above uses inputs.shape[dim].
  2. The centered sum of squares is torch.sum((inputs - self.sum / self.n) ** 2, dim=dim). self.sum has the reduced shape, so the subtraction only broadcasts when dim is the leading dimension.

As a result, for any dim other than 0 the call either raises a broadcasting RuntimeError, or, when the size of dim happens to equal the size of the leading dimension, it runs through and silently returns a wrong variance. The silent case is the worrying one because it is the shape you get from a square ensemble by time grid.

_update_var has the same broadcasting problem for a batch_dim other than 0 because it computes inputs - temp_sum / temp_n with the already reduced temp_sum.

The existing test_means_var only exercises dim=0 and is skipped when CUDA is not available, so CI never saw this.

I expected Variance(...)(x, dim=d) to match torch.var(x, dim=d) for every d, the same way Mean(...)(x, dim=d) matches torch.mean(x, dim=d).

Minimum reproducible example
import torch
import physicsnemo.metrics.general.ensemble_metrics as em

# Broadcasting error
x = torch.randn(4, 6, 5)
em.Variance((4, 5))(x, dim=1)
# RuntimeError: The size of tensor a (6) must match the size of tensor b (4) at non-singleton dimension 1

# Silent wrong result when the sizes happen to match
x = torch.randn(6, 6, 5)
var = em.Variance((6, 5))(x, dim=1)
print(torch.allclose(var, torch.var(x, dim=1)))
# False

# Mean is fine on the same input
print(torch.allclose(em.Mean((6, 5))(x, dim=1), torch.mean(x, dim=1)))
# True
Relevant log output
RuntimeError: The size of tensor a (6) must match the size of tensor b (4) at non-singleton dimension 1
Environment details
Bare-metal, CPU only, Python 3.12, torch CPU wheel, physicsnemo installed from source with pip install -e .

I have a fix ready with a CPU test that covers dim 1, 2 and -1 for both the error case and the silent case, and I will open a PR that references this issue.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in physicsnemo.metrics.general.ensemble_metrics at Variance.call, comparing its dimension handling with Mean.call; also inspect _update_var for the related batch_dim case. Run the existing test_means_var, then extend CPU coverage for dimensions 1, 2, and -1, including mismatched and equal leading-dimension sizes. Done means variance matches torch.var for each covered dimension.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.