pytorch / pytorch/TensorRT

[Bug] All-zero logits from convert_exported_program_to_serialized_trt_engine with use_explicit_typing=False

Open
#4,141 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Story: Runtime & Memory & Serialization
Dominant language
Python
Stars
3k
Forks
410
Avg merge
3d 18h
Merged PRs (30d)
78

Description

Bug description

convert_exported_program_to_serialized_trt_engine produces a TRT engine with all-zero logits when use_explicit_typing=False and enabled_precisions={torch.float16}. The same exported program converts correctly with use_explicit_typing=True.

No error or warning is emitted — the engine silently produces incorrect results.

To reproduce

import torch
import torch_tensorrt
from transformers import AutoModelForCausalLM, StaticCache

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B", dtype=torch.float16, device_map="cuda")
model.eval()

# Export with stateless KV cache wrapper
# (Standard torch.export flow — model has fp16 weights, fp32 I/O)
# ... export model to ExportedProgram ...

# This produces all-zero logits:
engine_bad = torch_tensorrt.dynamo.convert_exported_program_to_serialized_trt_engine(
    exported_program,
    inputs=example_inputs,
    use_explicit_typing=False,
    enabled_precisions={torch.float16},
    min_block_size=1,
    truncate_double=True,
)

# This produces correct logits:
engine_good = torch_tensorrt.dynamo.convert_exported_program_to_serialized_trt_engine(
    exported_program,
    inputs=example_inputs,
    use_explicit_typing=True,
    min_block_size=1,
    truncate_double=True,
)

Observed behavior

  • use_explicit_typing=False: Engine builds successfully (~1139 MB), but inference produces logits where every value is exactly 0.0
  • use_explicit_typing=True: Engine builds successfully (~1139 MB) with correct non-zero logits

Both engines are the same size, suggesting TRT builds a valid-looking engine in both cases, but the use_explicit_typing=False path silently corrupts computation — likely due to incorrect type casting or layer fusion.

Expected behavior

Both paths should produce correct (non-zero) logits, or use_explicit_typing=False should raise an error/warning if it cannot handle the exported program correctly.

Environment

  • torch: 2.10.0+cu128
  • torch_tensorrt: 2.10.0+cu130
  • tensorrt: 10.14.1.48
  • transformers: 5.2.0
  • Model: Qwen/Qwen3-0.6B (fp16, 28 layers, GQA with 16 heads / 2 KV heads)
  • GPU: NVIDIA GeForce RTX 4090 (24GB)
  • Driver: 590.48.01
  • OS: Linux (Ubuntu 24.04)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at torch_tensorrt.dynamo.convert_exported_program_to_serialized_trt_engine and reproduce the Qwen/Qwen3-0.6B case with use_explicit_typing=False and enabled_precisions={torch.float16}. Compare it with the explicit-typing path and inspect the generated engine's logits. Done means the non-explicit path produces correct non-zero logits or reports that the configuration is unsupported.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
compilers, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.