[TorchAO][UT] test/test_low_bit_optim.py AssertionError: Tensor-likes are not close!

### 🐛 Describe the bug

```
git clone https://github.com/pytorch/ao.git
cd ao/
python setup.py install
pip install pytest expecttest
pytest -v test/test_low_bit_optim.py::TestQuantize::test_bf16_stochastic_round_device_xpu_compile_False
```
```
_______________________________ TestQuantize.test_bf16_stochastic_round_device_xpu_compile_False ________________________________

self = , device = 'xpu'
compile = False

@parametrize("device", _DEVICES)
@parametrize("compile", [False, True])
def test_bf16_stochastic_round(self, device, compile):
x = torch.rand(32, device=device) * 100
x_rep = x.view(-1, 1).repeat(1, 100_000)

func = torch.compile(
_fp32_to_bf16_sr, fullgraph=True, dynamic=False, disable=not compile
)
x_rep_bf16 = func(x_rep)
assert x_rep_bf16.dtype is torch.bfloat16

# must cast BF16 tensor back to FP32 so that .mean() is accurate
> torch.testing.assert_close(x_rep_bf16.float().mean(1), x, atol=3e-5, rtol=3e-5)
E AssertionError: Tensor-likes are not close!
E
E Mismatched elements: 1 / 32 (3.1%)
E Greatest absolute difference: 0.00238037109375 at index (24,) (up to 3e-05 allowed)
E Greatest relative difference: 3.565528822946362e-05 at index (24,) (up to 3e-05 allowed)
E
E To execute this test, run the following from the base repo dir:
E python test/test_low_bit_optim.py TestQuantize.test_bf16_stochastic_round_device_xpu_compile_False
E
E This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
```

### Versions

torch 2.10.0.dev20251210+xpu
torchao https://github.com/zxd1997066/ao/tree/xiangdong/test1