[CI Breakage] More than 10 PRs failing due to E2E runtime test failures - "torch._dynamo.exc.InternalTorchDynamoError: TimeoutError: Timeout"

Nobody has claimed this yet.

There are many fx_importer tests failing with the error below. Various pull requests have been affected starting at least from this one:

https://github.com/llvm/torch-mlir/pull/4186

torch._dynamo.exc.InternalTorchDynamoError: TimeoutError: Timeout

FAIL - "AdaptiveMaxPool3dDynamicWithIndices_basic"
Runtime error: Traceback (most recent call last):
File "/home/runner/_work/torch-mlir/torch-mlir/build/tools/torch-mlir/python_packages/torch_mlir/torch_mlir_e2e_test/framework.py", line 354, in compile_and_run_test
trace = config.run(compiled, golden_trace)
File "/home/runner/_work/torch-mlir/torch-mlir/build/tools/torch-mlir/python_packages/torch_mlir/torch_mlir_e2e_test/configs/fx_importer_backend.py", line 61, in run
self._export_run(artifact, trace)
File "/home/runner/_work/torch-mlir/torch-mlir/build/tools/torch-mlir/python_packages/torch_mlir/torch_mlir_e2e_test/configs/fx_importer_backend.py", line 137, in _export_run
prog: ExportedProgram = torch.export.export(
File "/home/runner/.local/lib/python3.10/site-packages/torch/export/__init__.py", line 319, in export
raise e
File "/home/runner/.local/lib/python3.10/site-packages/torch/export/__init__.py", line 286, in export
return _export(
File "/home/runner/.local/lib/python3.10/site-packages/torch/export/_trace.py", line 1153, in wrapper
raise e
File "/home/runner/.local/lib/python3.10/site-packages/torch/export/_trace.py", line 1119, in wrapper
ep = fn(*args, **kwargs)
File "/home/runner/.local/lib/python3.10/site-packages/torch/export/exported_program.py", line 123, in wrapper
return fn(*args, **kwargs)
File "/home/runner/.local/lib/python3.10/site-packages/torch/export/_trace.py", line 2175, in _export
ep = _export_for_training(
File "/home/runner/.local/lib/python3.10/site-packages/torch/export/_trace.py", line 1153, in wrapper
raise e
File "/home/runner/.local/lib/python3.10/site-packages/torch/export/_trace.py", line 1119, in wrapper
ep = fn(*args, **kwargs)
File "/home/runner/.local/lib/python3.10/site-packages/torch/export/exported_program.py", line 123, in wrapper
return fn(*args, **kwargs)
File "/home/runner/.local/lib/python3.10/site-packages/torch/export/_trace.py", line 2036, in _export_for_training
export_artifact = export_func(
File "/home/runner/.local/lib/python3.10/site-packages/torch/export/_trace.py", line 1410, in _strict_export
gm_torch_level = _export_to_torch_ir(
File "/home/runner/.local/lib/python3.10/site-packages/torch/export/_trace.py", line 800, in _export_to_torch_ir
gm_torch_level, _ = torch._dynamo.export(
File "/home/runner/.local/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1849, in inner
result_traced = opt_f(*args, **kwargs)
File "/home/runner/.local/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 381, in __call__
return super().__call__(*args, **kwargs)
File "/home/runner/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/runner/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
File "/home/runner/.local/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 749, in compile_wrapper
return fn(*args, **kwargs)
File "/home/runner/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/runner/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
File "/home/runner/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1562, in __call__
return self._torchdynamo_orig_callable(
File "/home/runner/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 648, in __call__
return _compile(
File "/home/runner/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1198, in _compile
raise InternalTorchDynamoError(
File "/home/runner/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1137, in _compile
guarded_code = compile_inner(code, one_graph, hooks, transform)
File "/home/runner/.local/lib/python3.10/site-packages/torch/_utils_internal.py", line 98, in wrapper_function
return function(*args, **kwargs)
File "/home/runner/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 816, in compile_inner
return _compile_inner(code, one_graph, hooks, transform)
File "/home/runner/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 967, in _compile_inner
check_fn = CheckFunctionManager(
File "/home/runner/.local/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 2787, in __init__
builder, guard_manager = self.build_guards(
File "/home/runner/.local/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 3045, in build_guards
guard.create(builder)
File "/home/runner/.local/lib/python3.10/site-packages/torch/_guards.py", line 360, in create
return self.create_fn(builder, self)
File "/home/runner/.local/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 2222, in SHAPE_ENV
clib = CppCodeCache.load(func_str)
File "/home/runner/.local/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 2591, in load
return cls.load_async(*args, **kwargs)()
File "/home/runner/.local/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 2457, in load_async
"vec_isa": pick_vec_isa(),
File "/home/runner/.local/lib/python3.10/site-packages/torch/_inductor/cpu_vec_isa.py", line 432, in pick_vec_isa
_valid_vec_isa_list: list[VecISA] = valid_vec_isa_list()
File "/home/runner/.local/lib/python3.10/site-packages/torch/_inductor/cpu_vec_isa.py", line 419, in valid_vec_isa_list
isa_list.extend(
File "/home/runner/.local/lib/python3.10/site-packages/torch/_inductor/cpu_vec_isa.py", line 419, in
isa_list.extend(
File "/home/runner/.local/lib/python3.10/site-packages/torch/_inductor/cpu_vec_isa.py", line 147, in __bool__
return self.__bool__impl(config.cpp.vec_isa_ok)
File "/home/runner/.local/lib/python3.10/site-packages/torch/_inductor/cpu_vec_isa.py", line 157, in __bool__impl
return self.check_build(VecISA._avx_code)
File "/home/runner/.local/lib/python3.10/site-packages/torch/_inductor/cpu_vec_isa.py", line 108, in check_build
with lock:
File "/home/runner/.local/lib/python3.10/site-packages/torch/utils/_filelock.py", line 25, in __enter__
result = super().__enter__()
File "/home/runner/.local/lib/python3.10/site-packages/filelock/_api.py", line 376, in __enter__
self.acquire()
File "/home/runner/.local/lib/python3.10/site-packages/filelock/_api.py", line 344, in acquire
time.sleep(poll_interval)
File "/home/runner/_work/torch-mlir/torch-mlir/build/tools/torch-mlir/python_packages/torch_mlir/torch_mlir_e2e_test/framework.py", line 316, in handle_timeout
raise TimeoutError(self.error_message)
torch._dynamo.exc.InternalTorchDynamoError: TimeoutError: Timeout

The following tests fail for similar reason:

ElementwiseDivScalarRoundingModeTruncModule_basic
ElementwiseClampMaxModule_basic
ElementwiseAtenLogicalNotOpPromoteModule_basic
ElementwiseAtenIsneginfOpModule_basic
ElementwiseAsinhModule_basic
BoolTensorHandleSignless_basic
BatchNorm1DModule_basic
AvgPool3dCountIncludePadFalse_basic
AtenMmQuint8_basic
AtenIntTensorCharDtypeModule_basic
ArgmaxModule_with_dim
AddmmModuleFloat_basic

Start by running the listed fx_importer E2E tests and inspect the timeout handling in torch_mlir_e2e_test/framework.py around line 316 and the export path in configs/fx_importer_backend.py around line 137. Compare the failures across the named tests; done means the affected tests complete without the reported TimeoutError.

Written by the indexing model from the issue text.