intel / intel/torch-xpu-ops

timm_models botnet26t_256/eca_halonext26ts/rexnet_100 got fail_accuracy on Linux PVC(1550)

Open
#4,424 1 comment 0 reactions 2 assignees Claimed by @weishi-deng View on GitHub
Accuracy hw: PVC test: e2e
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 9h
Merged PRs (30d)
112

Description

### πŸ› Describe the bug

There is fail_accuracy for timm_models botnet26t_256/eca_halonext26ts/rexnet_100 train amp_bf16 on Linux **PVC(1550) but passed on PVC(1100)**

## Reproducer
### Setup env
```bash
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/xpu
TORCH_COMMIT_ID=$(python -c 'import torch; print(torch.version.git_version)')
git clone https://github.com/pytorch/pytorch pytorch
cd pytorch
git checkout ${TORCH_COMMIT_ID}
pip install requests

pip install pandas psutil scipy pyyaml
TIMM_COMMIT_ID="$(cat .ci/docker/ci_commit_pins/timm.txt 2> /dev/null || echo 'v1.0.14')"
pip install git+https://github.com/huggingface/pytorch-image-models@${TIMM_COMMIT_ID}

```
### Run commandline
```bash
python benchmarks/dynamo/timm_models.py --accuracy -d xpu -n10 --backend=inductor --cold-start-latency --training --amp --amp-dtype bfloat16 --only botnet26t_256
python benchmarks/dynamo/timm_models.py --accuracy -d xpu -n10 --backend=inductor --cold-start-latency --training --amp --amp-dtype bfloat16 --only eca_halonext26ts
python benchmarks/dynamo/timm_models.py --accuracy -d xpu -n10 --backend=inductor --cold-start-latency --training --amp --amp-dtype bfloat16 --only rexnet_100
```
### ErrorLog
```bash
(pt2.13) gta@DUT1024PVC:~/kaileiyx/pytorch-debug$ python benchmarks/dynamo/timm_models.py --accuracy -d xpu -n10 --backend=inductor --cold-start-latency --training --amp --amp-dtype bfloat16 --only botnet26t_256
loading model: 0it [00:05, ?it/s]
xpu train botnet26t_256
W0723 02:44:35.254000 641523 site-packages/torch/_inductor/utils.py:3146] [2/0_1] get_device_tflops: no Triton fallback available for non-CUDA devices. Returning 0.0; roofline estimates will use memory bandwidth only.
E0723 02:45:34.797000 641523 site-packages/torch/_dynamo/utils.py:3706] RMSE (res-fp64): 0.01796, (ref-fp64): 0.00811 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.010000, use_larger_multiplier_for_smaller_tensor: 0
E0723 02:45:34.797000 641523 site-packages/torch/_dynamo/utils.py:3538] Accuracy failed for key name stages.3.0.conv3_1x1.bn.running_var
fail_accuracy

(pt2.13) gta@DUT1024PVC:~/kaileiyx/pytorch-debug$ python benchmarks/dynamo/timm_models.py --accuracy -d xpu -n10 --backend=inductor --cold-start-latency --training --amp --amp-dtype bfloat16 --only eca_halonext26ts
loading model: 0it [00:00, ?it/s]Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
WARNING:huggingface_hub.utils._http:Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
model.safetensors: downloading bytes: β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 41.1MB, 3.49MB/s
model.safetensors: reconstructing file: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 43.2MB / 43.2MB, 3.85MB/s
loading model: 0it [00:11, ?it/s]
xpu train eca_halonext26ts
W0723 02:47:04.784000 643586 site-packages/torch/_inductor/utils.py:3146] [2/0_1] get_device_tflops: no Triton fallback available for non-CUDA devices. Returning 0.0; roofline estimates will use memory bandwidth only.
E0723 02:48:08.084000 643586 site-packages/torch/_dynamo/utils.py:3706] RMSE (res-fp64): 0.00383, (ref-fp64): 0.00124 and shape=torch.Size([2048]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.010000, use_larger_multiplier_for_smaller_tensor: 0
E0723 02:48:08.085000 643586 site-packages/torch/_dynamo/utils.py:3538] Accuracy failed for key name stages.3.0.conv3_1x1.bn.running_mean
fail_accuracy

(pt2.13) gta@DUT1024PVC:~/kaileiyx/pytorch-debug$ python benchmarks/dynamo/timm_models.py --accuracy -d xpu -n10 --backend=inductor --cold-start-latency --training --amp --amp-dtype bfloat16 --only rexnet_100
loading model: 0it [00:00, ?it/s]Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
WARNING:huggingface_hub.utils._http:Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
model.safetensors: downloading bytes: β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 18.5MB, 1.65MB/s
model.safetensors: reconstructing file: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 19.4MB / 19.4MB, 1.79MB/s
loading model: 0it [00:13, ?it/s]
xpu train rexnet_100
E0723 02:51:23.606000 645433 site-packages/torch/_dynamo/utils.py:3706] RMSE (res-fp64): 0.00302, (ref-fp64): 0.00043 and shape=torch.Size([1044]). res.dtype: torch.float32, multiplier: 2.000000, tol: 0.010000, use_larger_multiplier_for_smaller_tensor: 0
E0723 02:51:23.607000 645433 site-packages/torch/_dynamo/utils.py:3538] Accuracy failed for key name features.15.conv_dw.bn.running_var
fail_accuracy
```

### Versions

```bash
(pt2.13) gta@DUT1024PVC:~/kaileiyx/pytorch-debug$ pip list |grep -E 'intel|torch|triton'
intel-cmplr-lib-rt 2026.0.0
intel-cmplr-lib-ur 2026.0.0
intel-cmplr-lic-rt 2026.0.0
intel-opencl-rt 2026.0.0
intel-openmp 2026.0.0
intel-pti 0.17.0
intel-sycl-rt 2026.0.0
torch 2.13.0+xpu
torchao 0.18.0+xpu
torchaudio 2.11.0+xpu
torchvision 0.28.0+xpu
triton-xpu 3.7.2

Intel GPU models onboard:
* Intel(R) Data Center GPU Max 1550
* Intel(R) Data Center GPU Max 1550
* Intel(R) Data Center GPU Max 1550
* Intel(R) Data Center GPU Max 1550
* Intel(R) Data Center GPU Max 1550
* Intel(R) Data Center GPU Max 1550
* Intel(R) Data Center GPU Max 1550
* Intel(R) Data Center GPU Max 1550
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.