torchvision manywheel py 3.11, cuda failure
Open
Nobody has claimed this yet.
high priority
triage review
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 7.3k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 13
Description
🐛 Describe the bug
Following nightly failure:
https://github.com/pytorch/vision/actions/runs/9111531485/job/25048879022
Started on 5/16 . Most likely introduced by one of the changes in:
https://github.com/pytorch/pytorch/commit/a86434a3cae4f6d82bdcd242d7c793381cc484f2
+ /__w/vision/vision/3/bin/conda run -p /__w/_temp/conda_environment_9111531485 python pytorch/vision/test/smoke_test.py
ERROR conda.cli.main_run:execute(41): `conda run python pytorch/vision/test/smoke_test.py` failed. (See above for error)
Downloading: "https://download.pytorch.org/models/resnet50-11ad3fa6.pth" to /github/home/.cache/torch/hub/checkpoints/resnet50-11ad3fa6.pth
torchvision: 0.19.0.dev20240516+cu121
torch.cuda.is_available: True
/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_inductor/compile_fx.py:133: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
warnings.warn(
Traceback (most recent call last):
File "/__w/vision/vision/pytorch/vision/test/smoke_test.py", line 103, in <module>
main()
File "/__w/vision/vision/pytorch/vision/test/smoke_test.py", line 96, in main
smoke_test_compile()
torch.ops.image._jpeg_version() = 62
Is torchvision usable? True
German shepherd (cpu): 37.6%
German shepherd (cuda): 37.6%
File "/__w/vision/vision/pytorch/vision/test/smoke_test.py", line 42, in smoke_test_compile
out = model(x)
^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 414, in _fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 1085, in catch_errors
return callback(frame, cache_entry, hooks, frame_state, skip=1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 926, in _convert_frame
result = inner_convert(
^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 464, in _convert_frame_assert
return _compile(
^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_utils_internal.py", line 74, in wrapper_function
return function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 807, in _compile
guarded_code = compile_inner(code, one_graph, hooks, transform)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_dynamo/utils.py", line 210, in time_wrapper
r = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 626, in compile_inner
out_code = transform_code_object(code, transform)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_dynamo/bytecode_transformation.py", line 1167, in transform_code_object
transformations(instructions, code_options)
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 178, in _fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 572, in transform
tracer.run()
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2234, in run
super().run()
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 884, in run
while self.step():
^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 799, in step
self.dispatch_table[inst.opcode](self, inst)
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2423, in RETURN_VALUE
self._return(inst)
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2408, in _return
self.output.compile_subgraph(
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 1084, in compile_subgraph
self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root)
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 1301, in compile_and_call_fx_graph
compiled_fn = self.call_user_compiler(gm)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_dynamo/utils.py", line 210, in time_wrapper
r = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 1392, in call_user_compiler
raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 1373, in call_user_compiler
compiled_fn = compiler_fn(gm, self.example_inputs())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_dynamo/repro/after_dynamo.py", line 127, in debug_wrapper
compiled_gm = compiler_fn(gm, example_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/__init__.py", line 1747, in __call__
return compile_fx(model_, inputs_, config_patches=self.config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 1492, in compile_fx
return aot_autograd(
^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_dynamo/backends/common.py", line 65, in compiler_fn
cg = aot_module_simplified(gm, example_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_functorch/aot_autograd.py", line 965, in aot_module_simplified
compiled_fn = create_aot_dispatcher_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_dynamo/utils.py", line 210, in time_wrapper
r = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_functorch/aot_autograd.py", line 686, in create_aot_dispatcher_function
compiled_fn = compiler_fn(
^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line [445](https://github.com/pytorch/vision/actions/runs/9111531485/job/25048879022#step:16:446), in aot_dispatch_autograd
compiled_fw_func = aot_config.fw_compiler(fw_module, adjusted_flat_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_dynamo/utils.py", line 210, in time_wrapper
r = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 1397, in fw_compiler_base
return inner_compile(
^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_dynamo/repro/after_aot.py", line 83, in debug_wrapper
inner_compiled_fn = compiler_fn(gm, example_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_inductor/debug.py", line 304, in inner
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_dynamo/utils.py", line 210, in time_wrapper
r = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 522, in compile_fx_inner
compiled_graph = fx_codegen_and_compile(
^^^^^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 818, in fx_codegen_and_compile
compiled_fn = graph.compile_to_fn()
^^^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_inductor/graph.py", line 1736, in compile_to_fn
return self.compile_to_module().call
^^^^^^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_dynamo/utils.py", line 210, in time_wrapper
r = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_inductor/graph.py", line 1680, in compile_to_module
mod = PyCodeCache.load_by_key_path(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_inductor/codecache.py", line 2576, in load_by_key_path
mod = _reload_python_module(key, path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_inductor/runtime/compile_tasks.py", line 44, in _reload_python_module
exec(code, mod.__dict__, mod.__dict__)
File "/tmp/torchinductor_root/fk/cfka6mgq6jdwbss6b6p7iptbdbl4bt3qmhxjfnhaxkgaqknx6hu7.py", line 2330, in <module>
async_compile.wait(globals())
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_inductor/codecache.py", line 3137, in wait
scope[key] = result.result()
^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/site-packages/torch/_inductor/codecache.py", line 2939, in result
self.future.result()
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/concurrent/futures/_base.py", line [456](https://github.com/pytorch/vision/actions/runs/9111531485/job/25048879022#step:16:457), in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/__w/_temp/conda_environment_9111531485/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
SyntaxError: unterminated string literal (detected at line 1) (<unknown>, line 1)
Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
You can suppress this exception and fall back to eager by setting:
import torch._dynamo
torch._dynamo.config.suppress_errors = True
Versions
2.4.0
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked GitHub Actions run and test/smoke_test.py, especially smoke_test_compile(), then compare the failure with the referenced PyTorch commit. Reproduce the nightly manywheel Python 3.11 CUDA job and trace the compiler failure. Done means the affected CUDA smoke test completes successfully in the nightly workflow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- build-system, ci-cd, computer-vision
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100