[distributed][tensor] test_random_ops.py torch._dynamo.exc.TorchRuntimeError: RuntimeError when making fake tensor call
- Dominant language
- Python
- Stars
- 113
- Forks
- 129
- Avg merge
- 5d 9h
- Merged PRs (30d)
- 112
Description
### 🐛 Describe the bug
please get wheels from https://github.com/intel/torch-xpu-ops/actions/runs/22754158369 or use or use gh download
```
gh run download 22754158369 --repo intel/torch-xpu-ops --name Torch-XPU-Wheel-1826-22754158369-1 --dir path --pattern "*.zip"
git clone -b distributed_2.11 https://github.com/daisyden/pytorch.git
cd pytorch
pip install -r requirements.txt
pip install pytest expecttest
pytest -v test/distributed/tensor/test_random_ops.py::DistTensorRandomOpCompileTest::test_compile_bernoulli
pytest -v test/distributed/tensor/test_random_ops.py::DistTensorRandomOpCompileTest::test_compile_bernoulli_float
pytest -v test/distributed/tensor/test_random_ops.py::DistTensorRandomOpCompileTest::test_compile_multiple_random_ops
pytest -v test/distributed/tensor/test_random_ops.py::DistTensorRandomOpCompileTest::test_compile_native_dropout
pytest -v test/distributed/tensor/test_random_ops.py::DistTensorRandomOpCompileTest::test_compile_normal_
pytest -v test/distributed/tensor/test_random_ops.py::DistTensorRandomOpCompileTest::test_compile_rand_like
pytest -v test/distributed/tensor/test_random_ops.py::DistTensorRandomOpCompileTest::test_compile_randint_like
pytest -v test/distributed/tensor/test_random_ops.py::DistTensorRandomOpCompileTest::test_compile_randn_like
pytest -v test/distributed/tensor/test_random_ops.py::DistTensorRandomOpCompileTest::test_compile_uniform_
```
```
Traceback (most recent call last):
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/unittest/case.py", line 59, in testPartExecutor
yield
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/unittest/case.py", line 591, in run
self._callTestMethod(testMethod)
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/unittest/case.py", line 549, in _callTestMethod
method()
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 831, in wrapper
self._join_processes(fn)
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1101, in _join_processes
self._check_return_codes(fn, elapsed_time)
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1141, in _check_return_codes
raise RuntimeError(error)
RuntimeError: Process 0 exited with error code 10 and exception:
Traceback (most recent call last):
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 986, in run_test
getattr(self, test_name)()
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 833, in wrapper
fn()
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3370, in wrapper
method(*args, **kwargs)
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/testing/_internal/distributed/_tensor/common_dtensor.py", line 902, in wrapper
raise e
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/testing/_internal/distributed/_tensor/common_dtensor.py", line 899, in wrapper
func(self, *args, **kwargs) # type: ignore[misc]
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 241, in wrapper
return func(*args, **kwargs)
File "/home/sdp/xiangdong/pytorch/test/distributed/tensor/test_random_ops.py", line 872, in test_compile_uniform_
self._test_compile_random_op(fn, device_mesh, placements=placements)
File "/home/sdp/xiangdong/pytorch/test/distributed/tensor/test_random_ops.py", line 790, in _test_compile_random_op
self._run_eager_and_compiled(fn, create_input, num_runs)
File "/home/sdp/xiangdong/pytorch/test/distributed/tensor/test_random_ops.py", line 733, in _run_eager_and_compiled
compiled_results, compiled_rng_states = self._run_with_seed(
File "/home/sdp/xiangdong/pytorch/test/distributed/tensor/test_random_ops.py", line 712, in _run_with_seed
result = fn(x)
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1020, in compile_wrapper
return fn(*args, **kwargs)
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 2340, in __call__
result = self._torchdynamo_orig_backend(
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 727, in __call__
result = _compile(
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1851, in _compile
guarded_code, tracer_output = compile_inner(code, one_graph, hooks)
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/_utils_internal.py", line 96, in wrapper_function
return function(*args, **kwargs)
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1495, in compile_inner
return _compile_inner(code, one_graph, hooks)
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1540, in _compile_inner
dynamo_output = compile_frame(
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1403, in compile_frame
bytecode, tracer_output = transform_code_object(code, transform)
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1626, in transform_code_object
tracer_output = transformations(instructions, code_options)
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1375, in transform
tracer_output = trace_frame(
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 341, in _fn
return fn(*args, **kwargs)
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 861, in trace_frame
run_tracer()
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 842, in run_tracer
tracer.run()
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1694, in run
while self.step():
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1360, in step
self.dispatch_table[inst.opcode](self, inst)
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 898, in wrapper
return inner_fn(self, inst)
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2629, in CALL_FUNCTION
self.call_function(fn, args, {})
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1261, in call_function
self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type]
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 1343, in call_function
return self.obj.call_method(tx, self.name, list(args), kwargs)
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 870, in call_method
result = wrap_fx_proxy(tx, proxy)
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 2990, in wrap_fx_proxy
return wrap_fx_proxy_cls(target_cls=TensorVariable, **kwargs)
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 3065, in wrap_fx_proxy_cls
out: VTTypeAlias = _wrap_fx_proxy(
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 3189, in _wrap_fx_proxy
example_value = get_fake_value(proxy.node, tx, allow_non_graph_fake=True)
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 3713, in get_fake_value
return _get_fake_value_impl(node, tx, allow_non_graph_fake)
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 3904, in _get_fake_value_impl
_wrap_graph_break_with_torch_runtime_err(
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 3702, in _wrap_graph_break_with_torch_runtime_err
raise exc.with_traceback(e.__traceback__) from None
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 3699, in _wrap_graph_break_with_torch_runtime_err
gb_fn()
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 3905, in
lambda: unimplemented(
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 633, in unimplemented
raise Unsupported(
torch._dynamo.exc.TorchRuntimeError: RuntimeError when making fake tensor call
Explanation: Dynamo failed to run FX node with fake tensors: call_method uniform_(*(DTensor(local_tensor=FakeTensor(..., device='xpu:0', size=(2, 8)), device_mesh=DeviceMesh((4,), 'xpu', stride=(1,)), placements=(Shard(dim=0),)), 0.0, 1.0), **{}): got AssertionError("Please convert all Tensors to FakeTensors first or instantiate FakeTensorMode with 'allow_non_fake_inputs'. Found in aten.to.dtype_layout(tensor([...], size=(16,), dtype=torch.uint8), dtype=torch.uint8, layout=torch.strided, device=device(type='xpu', index=0))")
Hint: Your code may result in an error when running in eager. Please double check that your code doesn't contain a similar error when actually running eager/uncompiled. You can do this by removing the `torch.compile` call, or by using `torch.compiler.set_stance("force_eager")`.
Developer debug context:
For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb4315.html
from user code:
File "/home/sdp/xiangdong/pytorch/test/distributed/tensor/test_random_ops.py", line 869, in fn
return x.uniform_(0.0, 1.0)
Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
To execute this test, run the following from the base repo dir:
python test/distributed/tensor/test_random_ops.py DistTensorRandomOpCompileTest.test_compile_uniform_
This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
```
### Versions
https://github.com/daisyden/pytorch/tree/distributed_2.11
Contributor guide
Assessment
This issue has not been assessed yet.