[Driver 26.19] PT2E accuracy drops across vision models
- Dominant language
- Python
- Stars
- 113
- Forks
- 128
- Avg merge
- 5d 9h
- Merged PRs (30d)
- 112
Description
## Summary
Driver 26.19 shows 21 PT2E accuracy drops compared with driver 25.40.
## Affected cases
| Model | Dtype | Baseline Top1 | Target Top1 | Delta |
| --- | --- | --- | --- | --- |
| alexnet | float32 | 56.556 | 1.834 | -54.722 |
| alexnet | int8 | 55.710 | 2.394 | -53.316 |
| mnasnet1_0 | float32 | 73.468 | 2.278 | -71.190 |
| mnasnet1_0 | int8 | 71.632 | 1.172 | -70.460 |
| mobilenet_v2 | float32 | 71.870 | 1.512 | -70.358 |
| mobilenet_v2 | int8 | 71.056 | 1.102 | -69.954 |
| mobilenet_v3_large | float32 | 74.054 | 11.030 | -63.024 |
| mobilenet_v3_large | int8 | 67.386 | 6.104 | -61.282 |
| resnet152 | float32 | 78.314 | 2.146 | -76.168 |
| resnet152 | int8 | 77.686 | 1.118 | -76.568 |
| resnet18 | float32 | 69.760 | 1.718 | -68.042 |
| resnet18 | int8 | 67.704 | 1.802 | -65.902 |
| resnet50 | float32 | 76.146 | 2.166 | -73.980 |
| resnet50 | int8 | 75.072 | 1.416 | -73.656 |
| resnext50_32x4d | float32 | 77.608 | 5.296 | -72.312 |
| resnext50_32x4d | int8 | 77.350 | 3.092 | -74.258 |
| shufflenet_v2_x1_0 | float32 | 69.356 | 3.762 | -65.594 |
| squeezenet1_1 | float32 | 58.188 | 0.476 | -57.712 |
| squeezenet1_1 | int8 | 46.166 | 0.258 | -45.908 |
| vgg16 | float32 | 71.586 | 2.842 | -68.744 |
| vgg16 | int8 | 69.334 | 2.736 | -66.598 |
## Reproducer
### Environment setup
```bash
docker run -it -e TZ=Aisa/Shanghai --device=/dev/mem --device=/dev/dri --group-add video --privileged --shm-size=8g intelgpu/ubuntu-24.04-lts2:2523.40 bash
curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR="/usr/local/bin" sh
uv venv myvenv --python 3.12 --clear
source myvenv/bin/activate
uv pip install pip setuptools wheel
uv pip install torch torchaudio torchvision --index-url https://download.pytorch.org/whl/xpu
pytorch_commit="$(python -c 'import torch; print(torch.version.git_version)')"
git clone https://github.com/pytorch/pytorch
cd pytorch
git checkout ${pytorch_commit}
uv pip install pip pandas psutil scipy requests
uv pip install -U numpy==1.26.4 accelerate
uv pip install -r .ci/docker/ci_commit_pins/huggingface-requirements.txt
uv pip install timm
# update torch
# gh --repo intel/torch-xpu-ops run download 26399638675 -p "Torch-XPU-Wheel-*"
uv pip uninstall torch torchaudio torchvision triton-xpu -y
uv pip install Torch-XPU-Wheel-*/*.whl
git clone -b main https://github.com/chuanqi129/inductor-tools pt2e-accuracy
dataset_dir="${PWD}/_datasets/imagenet"
if [ ! -d ${dataset_dir} ];then
rm -rf ${dataset_dir} && mkdir -p ${dataset_dir} && cd ${dataset_dir}
wget -O valprep.sh https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh
wget -q https://image-net.org/data/ILSVRC/2012/ILSVRC2012_img_val.tar
tar -xf ILSVRC2012_img_val.tar
bash valprep.sh
fi
cmd_line=" python pt2e-accuracy/scripts/modelbench/quant/inductor_quant_acc.py --device xpu --dataset_dir ${dataset_dir} "
```
### PT2E accuracy commands
Run fp32 and int8 for each affected model:
```bash
for model_name in alexnet mnasnet1_0 mobilenet_v2 mobilenet_v3_large resnet152 resnet18 resnet50 resnext50_32x4d shufflenet_v2_x1_0 squeezenet1_1 vgg16
do
${cmd_line} --model_list ${model_name} --is_fp32
${cmd_line} --model_list ${model_name}
done
```
## Source runs
- Target: https://github.com/intel/torch-xpu-ops/actions/runs/26264076598
- Baseline: https://github.com/intel/torch-xpu-ops/actions/runs/26267132121
Contributor guide
Research direction
Start by comparing the target and baseline workflow runs, then reproduce the affected models with pt2e-accuracy/scripts/modelbench/quant/inductor_quant_acc.py using the documented Docker and XPU setup. Investigate why both FP32 and INT8 results drop, and consider the issue done when the affected model accuracies no longer regress against the baseline.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python, pytorch
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100