pytorch / pytorch/TensorRT

🐛 [Bug] run llm exmples are broken in thor platform

Open
#4,069 1 comment 0 reactions 1 assignee View on GitHub

@lanluo-nvidia is already working on this.

Since Feb 4, 2026.

bug story: LLM & Generative AI
Dominant language
Python
Stars
3k
Forks
410
Avg merge
3d 18h
Merged PRs (30d)
78

Description

Bug Description

Trying to export the model using torch.export.export()..
Trying torch.export._trace._export to trace the graph since torch.export.export() failed
WARNING:py.warnings:/opt/conda/envs/py310/lib/python3.10/copyreg.py:101: FutureWarning: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead.
  return cls.__new__(cls, *args)

WARNING:py.warnings:/workspace/TensorRT/tools/llm/cache_utils.py:126: FutureWarning: `treespec.children_specs` is deprecated. Use `treespec.child(index)` to access a single child, or `treespec.children()` to get all children.
  in_spec_for_args = in_spec.children_specs[0]

WARNING:py.warnings:/workspace/TensorRT/tools/llm/cache_utils.py:134: FutureWarning: `treespec.children_specs` is deprecated. Use `treespec.child(index)` to access a single child, or `treespec.children()` to get all children.
  in_spec_for_args.children_specs.append(_LEAF_SPEC)

WARNING:py.warnings:/workspace/TensorRT/tools/llm/cache_utils.py:139: FutureWarning: `treespec.children_specs` is deprecated. Use `treespec.child(index)` to access a single child, or `treespec.children()` to get all children.
  for child_spec in spec.children_specs:

Traceback (most recent call last):
  File "/workspace/TensorRT/tools/llm/run_llm.py", line 323, in <module>
    trt_model = compile_torchtrt(model, input_ids, args)
  File "/workspace/TensorRT/tools/llm/run_llm.py", line 134, in compile_torchtrt
    trt_model = torch_tensorrt.dynamo.compile(
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch_tensorrt/dynamo/_compiler.py", line 764, in compile
    gm = post_lowering(gm, settings)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch_tensorrt/dynamo/lowering/passes/_aten_lowering_pass.py", line 139, in post_lowering
    fake_tensor_updater.incremental_update(fake_mode)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch_tensorrt/dynamo/lowering/passes/_FakeTensorUpdater.py", line 193, in incremental_update
    new_fake_tensor = node.target(*args, **kwargs)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/_ops.py", line 819, in __call__
    return self._op(*args, **kwargs)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/utils/_stats.py", line 29, in wrapper
    return fn(*args, **kwargs)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1397, in __torch_dispatch__
    return self.dispatch(func, types, args, kwargs)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 2155, in dispatch
    return self._cached_dispatch_impl(func, types, args, kwargs)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1548, in _cached_dispatch_impl
    entry = self._make_cache_entry(state, key, func, args, kwargs, output)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1969, in _make_cache_entry
    output_info = self._get_output_info_for_cache_entry(
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1863, in _get_output_info_for_cache_entry
    synth_output = self._output_from_cache_entry(
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 2076, in _output_from_cache_entry
    return self._get_output_tensor_from_cache_entry(
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 2031, in _get_output_tensor_from_cache_entry
    empty = torch.empty_strided(
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/fx/experimental/sym_node.py", line 562, in expect_true
    return self.shape_env.guard_or_defer_runtime_assert(
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/fx/experimental/recording.py", line 273, in wrapper
    return retlog(fn(*args, **kwargs))
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 7779, in guard_or_defer_runtime_assert
    static_expr = self._maybe_evaluate_static(expr)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 2636, in wrapper
    return fn_cache(self, *args, **kwargs)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 6437, in _maybe_evaluate_static
    r = _maybe_evaluate_static_worker(
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 2421, in _maybe_evaluate_static_worker
    assert vr is not None
AssertionError

To Reproduce

Steps to reproduce the behavior:

Expected behavior

Environment

Build information about Torch-TensorRT can be found by turning on debug messages

  • Torch-TensorRT Version (e.g. 1.0.0):
  • PyTorch Version (e.g. 1.0):
  • CPU Architecture:
  • OS (e.g., Linux):
  • How you installed PyTorch (conda, pip, libtorch, source):
  • Build command you used (if compiling from source):
  • Are you using local sources or building from archives:
  • Python version:
  • CUDA version:
  • GPU models and configuration:
  • Any other relevant information:

Additional context

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.