[2.9][BMG-Windows][Torchbench] speeach_transforer accuracy_training failed with Exception Code: 0xC0000005 when using torchbench pinned by pytorch2.8
- Dominant language
- Python
- Stars
- 113
- Forks
- 128
- Avg merge
- 5d 13h
- Merged PRs (30d)
- 107
Description
### 🐛 Describe the bug
On BMG-Windows, the torchbench model - speeach_transforer accuracy_training got _**Exception Code: 0xC0000005**_ when using torchbench pinned by pytorch2.8. (Using the torchbench version pinned by 2.9, there would be an "_**OSError:** Could not load this library: C:\conda_env\Lib\site-packages\fbgemm_gpu\[**fbgemm_gpu_py.so**](http://fbgemm_gpu_py.so/"_.
Reproducer
```
conda create -n pt29 python=3.10 -y
conda activate pt29
pip install --pre torch==2.9.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/test/xpu
cd pytorch
pip install -r requirements.txt
pip install -r .ci\docker\requirements-ci.txt
pip install pytest-timeout
pip install --force-reinstall git+https://github.com/huggingface/transformers@v4.54.0
pip install -U pandas==2.2.3 psutil scipy numpy==1.26.4
pip install --no-deps git+https://github.com/huggingface/pytorch-image-models@5d535d7a2d4b435b1b5c1177fd8f04a12b942b9a
git clone https://github.com/pytorch/benchmark.git
cd benchmark
pip install --no-deps -r requirements.txt
python install.py --continue_on_fail
pip install onnx boto3 doctr gym fastNLP higher opacus dominate segment_anything_fast kaldi_io effdet pycocotools opencv-python torch_geometric tensorboardX
pip install pyre_extensions torchrec torch_geometric
pip install torchmultimodal
# TB models
python benchmarks\dynamo\torchbench.py --accuracy -d xpu -n10 --backend=inductor --cold-start-latency --training --bfloat16 --only speech_transformer
```
Error log
```
C:\pytorch>python benchmarks\dynamo\torchbench.py --accuracy -d xpu -n10 --backend=inductor --cold-start-latency --training --bfloat16 --only speech_transformer
C:\.conda\envs\pt29\lib\site-packages\torch\cuda\__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
loading model: 0it [00:00, ?it/s]################################################################################
### WARNING, path does not exist: KALDI_ROOT=/tmp
### (please add 'export KALDI_ROOT=' in your $HOME/.profile)
### (or run as: KALDI_ROOT= python .py)
################################################################################
loading model: 0it [00:03, ?it/s]
xpu train speech_transformer
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to DeviceCopy ops
cudagraph partition into 2 partitions
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to DeviceCopy ops
cudagraph partition into 2 partitions
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to non gpu ops
cudagraph partition due to DeviceCopy ops
cudagraph partition into 2 partitions
Exception Code: 0xC0000005
#0 0x00007ff8f408161d (C:\Users\sdp\AppData\Local\Temp\tmp191k8qjf\rg\crgm2qh5ndl2zbeae36cqmzemw7snxesg2clkx3le633lxfmxdnh.main.pyd+0x161d)
#1 0x00007ff8e8ce4d98 cfunction_call \Objects\methodobject.c:554:0
#2 0x00007ff8e8c9efeb _PyObject_MakeTpCall \Objects\call.c:215:0
#3 0x00007ff8e8daa812 call_function \Python\ceval.c:5899:0
#4 0x00007ff8e8da6687 _PyEval_EvalFrameDefault \Python\ceval.c:4214:0
#5 0x00007ff8e8da8e6b _PyEval_Vector \Python\ceval.c:5074:0
#6 0x00007ff8e8c9f3ae _PyFunction_Vectorcall \Objects\call.c:347:0
#7 0x00007ff8e8ca1691 _PyObject_VectorcallTstate \Include\cpython\abstract.h:114:0
#8 0x00007ff8e8ca184d method_vectorcall \Objects\classobject.c:53:0
#9 0x00007ff8e8da1da1 _PyObject_VectorcallTstate \Include\cpython\abstract.h:114:0
#10 0x00007ff8e8daa812 call_function \Python\ceval.c:5899:0
#11 0x00007ff8e8da6687 _PyEval_EvalFrameDefault \Python\ceval.c:4214:0
#12 0x00007ff8e8da8e6b _PyEval_Vector \Python\ceval.c:5074:0
#13 0x00007ff8e8c9f3ae _PyFunction_Vectorcall \Objects\call.c:347:0
#14 0x00007ff8e8da1da1 _PyObject_VectorcallTstate \Include\cpython\abstract.h:114:0
#15 0x00007ff8e8daa812 call_function \Python\ceval.c:5899:0
#16 0x00007ff8e8da6ca9 _PyEval_EvalFrameDefault \Python\ceval.c:4183:0
#17 0x00007ff8e8da8e6b _PyEval_Vector \Python\ceval.c:5074:0
#18 0x00007ff8e8c9f3ae _PyFunction_Vectorcall \Objects\call.c:347:0
#19 0x00007ff8e8c9edf4 _PyObject_FastCallDictTstate \Objects\call.c:142:0
#20 0x00007ff8e8c9f625 _PyObject_Call_Prepend \Objects\call.c:431:0
#21 0x00007ff8e8d0f494 slot_tp_call \Objects\typeobject.c:7495:0
#22 0x00007ff8e8c9f2a7 _Py_LeaveRecursiveCall \Include\internal\pycore_ceval.h:100:0
#23 0x00007ff8e8c9f2a7 _PyObject_Call \Objects\call.c:307:0
#24 0x00007ff8e8daaa24 do_call_core \Python\ceval.c:5945:0
#25 0x00007ff8e8da59c0 _PyEval_EvalFrameDefault \Python\ceval.c:4278:0
#26 0x00007ff8e8da8e6b _PyEval_Vector \Python\ceval.c:5074:0
#27 0x00007ff8e8c9f3ae _PyFunction_Vectorcall \Objects\call.c:347:0
#28 0x00007ff8e8da1da1 _PyObject_VectorcallTstate \Include\cpython\abstract.h:114:0
#29 0x00007ff8e8daa812 call_function \Python\ceval.c:5899:0
#30 0x00007ff8e8da6687 _PyEval_EvalFrameDefault \Python\ceval.c:4214:0
#31 0x00007ff8e8da8e6b _PyEval_Vector \Python\ceval.c:5074:0
#32 0x00007ff8e8c9f3ae _PyFunction_Vectorcall \Objects\call.c:347:0
#33 0x00007ff8e8da1da1 _PyObject_VectorcallTstate \Include\cpython\abstract.h:114:0
#34 0x00007ff8e8daa812 call_function \Python\ceval.c:5899:0
#35 0x00007ff8e8da6d21 _PyEval_EvalFrameDefault \Python\ceval.c:4232:0
#36 0x00007ff8e8da8e6b _PyEval_Vector \Python\ceval.c:5074:0
#37 0x00007ff8e8c9f3ae _PyFunction_Vectorcall \Objects\call.c:347:0
#38 0x00007ff8e8da1da1 _PyObject_VectorcallTstate \Include\cpython\abstract.h:114:0
#39 0x00007ff8e8daa812 call_function \Python\ceval.c:5899:0
#40 0x00007ff8e8da6ca9 _PyEval_EvalFrameDefault \Python\ceval.c:4183:0
#41 0x00007ff8e8da8e6b _PyEval_Vector \Python\ceval.c:5074:0
#42 0x00007ff8e8c9f3ae _PyFunction_Vectorcall \Objects\call.c:347:0
#43 0x00007ff8e8da1da1 _PyObject_VectorcallTstate \Include\cpython\abstract.h:114:0
#44 0x00007ff8e8daa812 call_function \Python\ceval.c:5899:0
#45 0x00007ff8e8da6687 _PyEval_EvalFrameDefault \Python\ceval.c:4214:0
#46 0x00007ff8e8da8e6b _PyEval_Vector \Python\ceval.c:5074:0
#47 0x00007ff8e8c9f3ae _PyFunction_Vectorcall \Objects\call.c:347:0
#48 0x00007ff8e8daaa24 do_call_core \Python\ceval.c:5945:0
#49 0x00007ff8e8da59c0 _PyEval_EvalFrameDefault \Python\ceval.c:4278:0
#50 0x00007ff8e8da8e6b _PyEval_Vector \Python\ceval.c:5074:0
#51 0x00007ff8e8c9f3ae _PyFunction_Vectorcall \Objects\call.c:347:0
#52 0x00007ff8e8ca1691 _PyObject_VectorcallTstate \Include\cpython\abstract.h:114:0
#53 0x00007ff8e8ca18ec method_vectorcall \Objects\classobject.c:83:0
#54 0x00007ff8ce2638bc torch::autograd::registerFunctionPreHook(struct torch::autograd::Node &, struct _object *) (C:\Users\sdp\.conda\envs\pt29_old_bench\lib\site-packages\torch\lib\torch_python.dll+0x4e38bc)
#55 0x00007ff8aa617b0a torch::autograd::Node::operator()(class std::vector> &&) (C:\Users\sdp\.conda\envs\pt29_old_bench\lib\site-packages\torch\lib\torch_cpu.dll+0x80d7b0a)
#56 0x00007ff8ab4d2cc1 torch::autograd::Engine::add_thread_pool_task(class std::weak_ptr const &) (C:\Users\sdp\.conda\envs\pt29_old_bench\lib\site-packages\torch\lib\torch_cpu.dll+0x8f92cc1)
#57 0x00007ff8ab4d41da torch::autograd::Engine::evaluate_function(class std::shared_ptr &, struct torch::autograd::Node *, struct torch::autograd::InputBuffer &, class std::shared_ptr const &) (C:\Users\sdp\.conda\envs\pt29_old_bench\lib\site-packages\torch\lib\torch_cpu.dll+0x8f941da)
#58 0x00007ff8ab4da2b2 torch::autograd::Engine::thread_main(class std::shared_ptr const &) (C:\Users\sdp\.conda\envs\pt29_old_bench\lib\site-packages\torch\lib\torch_cpu.dll+0x8f9a2b2)
#59 0x00007ff8ab4d9bbe torch::autograd::Engine::thread_init(int, class std::shared_ptr const &, bool) (C:\Users\sdp\.conda\envs\pt29_old_bench\lib\site-packages\torch\lib\torch_cpu.dll+0x8f99bbe)
#60 0x00007ff8ce256c8e torch::autograd::registerFunctionPreHook(struct torch::autograd::Node &, struct _object *) (C:\Users\sdp\.conda\envs\pt29_old_bench\lib\site-packages\torch\lib\torch_python.dll+0x4d6c8e)
#61 0x00007ff8ab4cd495 torch::autograd::Engine::get_base_engine(void) (C:\Users\sdp\.conda\envs\pt29_old_bench\lib\site-packages\torch\lib\torch_cpu.dll+0x8f8d495)
#62 0x00007ff9542037b0 (C:\WINDOWS\System32\ucrtbase.dll+0x37b0)
#63 0x00007ff95626e8d7 (C:\WINDOWS\System32\KERNEL32.DLL+0x2e8d7)
#64 0x00007ff956ce8d9c (C:\WINDOWS\SYSTEM32\ntdll.dll+0x8d9c)
```
### Versions
Versions
PyTorch Commit: 2.9.0+xpu / c31a8186c1a43e0cd24521e8fe18dc650959ca19
Platform: BMG
OS: Microsoft Windows 11 Pro
Driver: 32.0.101.7026
torchbench: 67b26a25a5211e5927ae72432ed95b47e83b83a2
Contributor guide
Research direction
Start by reproducing the reported Windows BMG failure with benchmarks\dynamo\torchbench.py using --only speech_transformer, --accuracy, --training, and --bfloat16 in the documented PyTorch 2.9 XPU environment. Compare the torchbench revisions and dependency versions described in the report, then use the command's completion without the access-violation crash as the verification point.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100