intel / intel/torch-xpu-ops

[Driver 26.19] PT2E accuracy drops across vision models

Open
#3,792 1 comment 0 reactions 0 assignees View on GitHub
dependency component: driver hw: PVC
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 9h
Merged PRs (30d)
112

Description

## Summary
Driver 26.19 shows 21 PT2E accuracy drops compared with driver 25.40.

## Affected cases
| Model | Dtype | Baseline Top1 | Target Top1 | Delta |
| --- | --- | --- | --- | --- |
| alexnet | float32 | 56.556 | 1.834 | -54.722 |
| alexnet | int8 | 55.710 | 2.394 | -53.316 |
| mnasnet1_0 | float32 | 73.468 | 2.278 | -71.190 |
| mnasnet1_0 | int8 | 71.632 | 1.172 | -70.460 |
| mobilenet_v2 | float32 | 71.870 | 1.512 | -70.358 |
| mobilenet_v2 | int8 | 71.056 | 1.102 | -69.954 |
| mobilenet_v3_large | float32 | 74.054 | 11.030 | -63.024 |
| mobilenet_v3_large | int8 | 67.386 | 6.104 | -61.282 |
| resnet152 | float32 | 78.314 | 2.146 | -76.168 |
| resnet152 | int8 | 77.686 | 1.118 | -76.568 |
| resnet18 | float32 | 69.760 | 1.718 | -68.042 |
| resnet18 | int8 | 67.704 | 1.802 | -65.902 |
| resnet50 | float32 | 76.146 | 2.166 | -73.980 |
| resnet50 | int8 | 75.072 | 1.416 | -73.656 |
| resnext50_32x4d | float32 | 77.608 | 5.296 | -72.312 |
| resnext50_32x4d | int8 | 77.350 | 3.092 | -74.258 |
| shufflenet_v2_x1_0 | float32 | 69.356 | 3.762 | -65.594 |
| squeezenet1_1 | float32 | 58.188 | 0.476 | -57.712 |
| squeezenet1_1 | int8 | 46.166 | 0.258 | -45.908 |
| vgg16 | float32 | 71.586 | 2.842 | -68.744 |
| vgg16 | int8 | 69.334 | 2.736 | -66.598 |

## Reproducer
### Environment setup
```bash
docker run -it -e TZ=Aisa/Shanghai --device=/dev/mem --device=/dev/dri --group-add video --privileged --shm-size=8g intelgpu/ubuntu-24.04-lts2:2523.40 bash

curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR="/usr/local/bin" sh
uv venv myvenv --python 3.12 --clear
source myvenv/bin/activate
uv pip install pip setuptools wheel
uv pip install torch torchaudio torchvision --index-url https://download.pytorch.org/whl/xpu

pytorch_commit="$(python -c 'import torch; print(torch.version.git_version)')"
git clone https://github.com/pytorch/pytorch
cd pytorch
git checkout ${pytorch_commit}

uv pip install pip pandas psutil scipy requests
uv pip install -U numpy==1.26.4 accelerate
uv pip install -r .ci/docker/ci_commit_pins/huggingface-requirements.txt
uv pip install timm

# update torch
# gh --repo intel/torch-xpu-ops run download 26399638675 -p "Torch-XPU-Wheel-*"
uv pip uninstall torch torchaudio torchvision triton-xpu -y
uv pip install Torch-XPU-Wheel-*/*.whl

git clone -b main https://github.com/chuanqi129/inductor-tools pt2e-accuracy
dataset_dir="${PWD}/_datasets/imagenet"
if [ ! -d ${dataset_dir} ];then
rm -rf ${dataset_dir} && mkdir -p ${dataset_dir} && cd ${dataset_dir}
wget -O valprep.sh https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh
wget -q https://image-net.org/data/ILSVRC/2012/ILSVRC2012_img_val.tar
tar -xf ILSVRC2012_img_val.tar
bash valprep.sh
fi
cmd_line=" python pt2e-accuracy/scripts/modelbench/quant/inductor_quant_acc.py --device xpu --dataset_dir ${dataset_dir} "
```

### PT2E accuracy commands
Run fp32 and int8 for each affected model:
```bash
for model_name in alexnet mnasnet1_0 mobilenet_v2 mobilenet_v3_large resnet152 resnet18 resnet50 resnext50_32x4d shufflenet_v2_x1_0 squeezenet1_1 vgg16

do
${cmd_line} --model_list ${model_name} --is_fp32
${cmd_line} --model_list ${model_name}
done
```

## Source runs
- Target: https://github.com/intel/torch-xpu-ops/actions/runs/26264076598
- Baseline: https://github.com/intel/torch-xpu-ops/actions/runs/26267132121

Contributor guide

Open the contributing guide

Research direction

Start by comparing the target and baseline workflow runs, then reproduce the affected models with pt2e-accuracy/scripts/modelbench/quant/inductor_quant_acc.py using the documented Docker and XPU setup. Investigate why both FP32 and INT8 results drop, and consider the issue done when the affected model accuracies no longer regress against the baseline.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python, pytorch
Domain
machine-learning, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.