NVIDIA / NVIDIA/TensorRT-Model-Connect

Docs: ITrtModule::forward() output-buffer lifetime and dtype contract are undocumented

Open Beginner friendly
#1,126 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
254
Forks
58
Avg merge
1d 7h
Merged PRs (30d)
235

Description

Request type

Add missing documentation

Documentation location
  • include/trtmc/runtime/trt_module.h — the forward() declaration
  • website/docs/architecture/runtime-lifecycle.md — "Runtime source map", which
    points at that header as the backend abstraction
Problem or missing content

ITrtModule::forward() carries two contracts that are not written down
anywhere, and both fail by producing plausible wrong output rather than an
error.

1. Returned tensors point into a per-module staging buffer that the next call
overwrites.
In src/runtime/backend/trt_module_impl.cpp:

auto& staging = host_output_staging_[name];
cudaMemcpy(staging.data(), entry.d_ptr, runtime_nbytes, cudaMemcpyDeviceToHost);

Tensor t;
t.data = staging.data();

host_output_staging_ is a member map keyed by tensor name
(src/runtime/backend/trt_module_impl.h). So a caller that holds a Tensor
from one forward() across a second forward() on the same module silently
reads the second call's data.

The declaration in include/trtmc/runtime/trt_module.h is grouped under a bare
// Forward passes comment and says nothing about this.

2. Outputs come back in the engine's own dtype (t.dtype = entry.dtype). A
bf16 engine returns half-width data, so a caller that allocates or reinterprets
as float32 gets garbage rather than a shape or type error.

Why this is worth a few lines in the header: it is a shared, model-agnostic
runtime contract, and the failure mode is silent. Concretely, in #1123
classifier-free guidance was a complete no-op for some time — the conditional
and unconditional branches ran through one module, the first pointer was still
held, and after the second pass both pointers referred to the unconditional
result. The pipeline ran, produced audio, and sounded plausible. It was found
only by measuring conditional_vs_unconditional rms == 0, not by reading the
code. Any family that runs two branches through one module and compares them is
exposed to the same thing.

Verification and search performed
grep -rn "host_output_staging" website/docs plugins   # 0 results
grep -rn "output buffer" website/docs                 # 0 results
grep -n "forward\|lifetime\|reuse\|buffer" include/trtmc/runtime/trt_module.h
# -> only the declarations and an unrelated "Direct buffer access (KV cache binding)"

grep -rn "next forward\|reused across\|alias" src/runtime/models/*/pipeline.cpp
# -> 2 hits, both in the single family added by #1123, both written after hitting the bug

Checked and found not to cover it: website/docs/architecture/runtime-lifecycle.md,
website/docs/architecture/runtime-plugins.md, the fp16-trt-network skill
(covers dtype inside the network, not on the runtime output side), and
website/docs/wiki/TRT-Internals.md (an archived redirect stub — this is not a
request to restore it).

Suggested correction

A short comment block above the forward() declarations in
include/trtmc/runtime/trt_module.h, stating that returned Tensor::data
points into per-module storage reused by the next call on that module, that a
caller must copy anything it needs to keep across calls, and that outputs carry
the engine's dtype and may be half-width. Optionally one cross-referencing
sentence in runtime-lifecycle.md, which already names this header.

Scoping note: this describes the host forward() path. forward_device() and
externally bound buffers behave differently, so the wording should say which
path it covers — a maintainer confirming the intended contract is more useful
here than my inferring it from the implementation, which is why this is an issue
rather than a PR.

Happy to send the patch if the wording direction looks right.

Submission checks
  • I searched open and closed documentation issues and found no duplicate.
  • I removed secrets, private/internal evidence, personal paths, and restricted artifacts.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the forward() declarations in include/trtmc/runtime/trt_module.h and compare them with src/runtime/backend/trt_module_impl.cpp and its header. Check website/docs/architecture/runtime-lifecycle.md for the existing source-map wording. Done means the host forward() contract documents output-buffer reuse, required copying, engine dtype, and its scope, with any agreed cross-reference added to the lifecycle page.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
82/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.