intel / intel/torch-xpu-ops

[release/2.12][PVC] BERT_pytorch amp_bf16 inference got fail_accuracy

Open
#3,463 1 comment 0 reactions 1 assignee Claimed by @CuiYifeng View on GitHub
Accuracy hw: PVC test: e2e
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 9h
Merged PRs (30d)
112

Description

### 🐛 Describe the bug

torchbench BERT_pytorch got `fail_accuracy` with amp_bf16 inference on PVC with PyTorch 2.12 RC wheel.

Accuracy verification confirmed this is a real failure (not flaky).

### Error Log

```
2026-04-24 05:43:26 [INFO] [1/10] Running BenchmarkSpec(suite='torchbench', dtype='amp_bf16', mode='inference', scenario='accuracy', model='BERT_pytorch')
2026-04-24 05:43:26 [INFO] Checking: torch and triton
2026-04-24 05:43:26 [DEBUG] Running: /workspace/myvenv/bin/python3 -c 'import torch; print(torch.version.git_version)'
2026-04-24 05:43:27 [DEBUG] Running: uv pip list --python /workspace/myvenv/bin/python3 2>&1 |grep triton-xpu |tail -n 1 |sed 's/.* //'
2026-04-24 05:43:27 [INFO] Torch: 489deee763382be6f9ccb296797b782d7054a06a, Triton: 3.7.1
2026-04-24 05:43:27 [DEBUG] Running: sudo rm -rf ~/.triton/ /tmp/torch* /tmp/tmp*
2026-04-24 05:43:27 [DEBUG] Running: /workspace/myvenv/bin/python3 /workspace/pytorch-src/benchmarks/dynamo/torchbench.py --accuracy --amp --amp-dtype bfloat16 --inference -d xpu -n 10 --only BERT_pytorch --backend=inductor --cold-start-latency --timeout 10800 --disable-cudagraphs --ou
tput /workspace/inductor_log/489deee763382be6f9ccb296797b782d7054a06a/torchbench/amp_bf16/inductor_torchbench_amp_bf16_inference_xpu_1777009406.284249_accuracy.csv 2>&1 | tee -a /workspace/inductor_log/489deee763382be6f9ccb296797b782d7054a06a/torchbench/amp_bf16/inductor_torchbench_amp
_bf16_inference_xpu_1777009406.284249_accuracy.log
loading model: 0it [00:02, ?it/s]
xpu eval BERT_pytorch
E0424 05:43:56.662000 194101 torch/_dynamo/utils.py:3519] RMSE (res-fp64): 0.00196, (ref-fp64): 0.00058 and shape=torch.Size([4, 2]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000, use_larger_multiplier_for_smaller_tensor: 0
fail_accuracy

```

Torch commit: 489deee763382be6f9ccb296797b782d7054a06a
Triton: 3.7.1

### Reproducer

```bash
docker run -it -e TZ=Asia/Shanghai --device=/dev/mem --device=/dev/dri \
--group-add video --privileged --shm-size=8g intelgpu/ubuntu-24.04-lts2:2523.40 bash

curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR="/usr/local/bin" sh
uv venv myvenv --python 3.12 --clear
source myvenv/bin/activate
uv pip install pip numpy 'setuptools<81' wheel

uv pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/test/xpu

pytorch_commit="$(python -c 'import torch; print(torch.version.git_version)')"
git clone https://github.com/pytorch/pytorch && cd pytorch && git checkout ${pytorch_commit}
rm -rf torch torch-xpu-ops
git clone https://github.com/intel/torch-xpu-ops
rsync -avz ./torch-xpu-ops/.ci/benchmarks/ ./benchmarks/dynamo/

uv pip install pandas psutil scipy requests
uv pip install -U numpy==1.26.4

git clone https://github.com/pytorch/benchmark tmp-benchmark && cd tmp-benchmark
uv pip install -r requirements.txt && python install.py BERT_pytorch && cd ..

python ./benchmarks/dynamo/torchbench.py --accuracy --amp --amp-dtype bfloat16 \
--inference -d xpu -n 10 --only BERT_pytorch --backend=inductor \
--cold-start-latency --timeout 10800 --disable-cudagraphs
```

### Versions

- PyTorch 2.12 RC (torch commit: 489deee763382be6f9ccb296797b782d7054a06a)
- Triton: 3.7.1
- Platform: PVC (Intel Data Center GPU Max)
- Docker: intelgpu/ubuntu-24.04-lts2:2523.40

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.