NVIDIA / NVIDIA/TensorRT-LLM

An absorbed forward step returning None crashes the PyTorch executor loop

Open
#17,913 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Pytorch
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

System Info
  • TensorRT-LLM: 1.3.0rc23 (built from f20ea652); behaviour re-checked against main @ fbda11f5
  • GPU: NVIDIA GeForce RTX 5090 (sm_120), driver 595.71.05
  • PyTorch 2.11.0+cu130, CUDA 13.0, Python 3.12.13
  • Ubuntu 25.10, kernel 6.17.0-41-generic
  • Backend: PyTorch; single GPU, no TP/PP
  • Model where observed: Qwen3.6-35B-A3B-NVFP4 with FP8 KV cache (where model-specific)
Who can help?

No response

Reproduction

_forward_step() can return None when a forward pass is absorbed rather than producing outputs.
The executor loop then dereferences that None, and the exception escapes the loop — taking down the
executor rather than skipping the affected iteration.

This affects both the overlap and non-overlap PyTorch executor paths, which is why the fix touches
both.

The executor loop is process-wide: one absorbed forward step ends serving for every in-flight and
subsequent request, not just the one that triggered it. The HTTP frontend can remain up while this
has happened, so it presents as requests failing against an apparently healthy server.

Expected behavior

An absorbed forward step is treated as 'no outputs this iteration' and serving continues.

actual behavior

The None result is dereferenced and the exception escapes the executor loop, ending serving for every in-flight and subsequent request. The HTTP frontend can stay up while this has happened.

additional notes

Treat a None result as "no outputs this iteration" and continue, rather than dereferencing it. The
added branch is exercised only when _forward_step() returns None; normal forward and sampling
behaviour is unchanged.

Deliberately out of scope: the inter-pipeline-parallel path, where recovery from a failed forward
must be coordinated across ranks rather than handled by a local skip.

We have the patch and a CPU-only regression test that fails before and passes after, and are raising
this Issue Request first per CONTRIBUTING.md.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the PyTorch executor loop and its _forward_step() call, checking both the overlap and non-overlap paths. Run the CPU-only regression test mentioned in the issue; done means a None result is treated as no outputs, the iteration is skipped, and normal serving continues without changing regular forward or sampling behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
ai, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.