[Feature]: Zero-initialize additionalOutput buffers to prevent stale data in streaming mode

🚀 The feature, motivation and pitch

Motivation

We are using TensorRT-LLM for our proprietary model in a text-to-speech (TTS) pipeline. Instead of using generation tokens directly, we retrieve generation hidden_states via the additionalOutput API to feed into our speech synthesis module.

While developing streaming inference, we discovered that the additionalOutput buffers (specifically mAdditionalGenerationOutputTensors) retain data from previous requests.

Problem scenario:

1. Run inference Request A (e.g., 100 tokens generated)
2. Run inference Request B (e.g., 50 tokens generated)
3. Run inference Request C (e.g., 30 tokens generated)
4. Issue: Request C's hidden_states buffer still contains leftover data from Request A at positions 30-99

Why this matters for streaming

• In non-streaming mode: token count and hidden_states length are always synchronized, so slicing by token count works fine
• In streaming mode: tokens and hidden_states are filled at different rates, making it impossible to reliably slice by token count

Root Cause

In cpp/include/tensorrt_llm/batch_manager/llmRequest.h, the allocAdditionalOutputs function uses pinnedPool which reuses memory from a MemoryPool without clearing it:

auto tensor = runtime::BufferManager::pinnedPool(shape, dataType);
// No zero-initialization here
outputTensor.second = std::move(tensor);

Feature Request

Add zero-initialization for mAdditionalGenerationOutputTensors in allocAdditionalOutputs:

auto tensor = runtime::BufferManager::pinnedPool(shape, dataType);
std::memset(tensor->data(), 0, tensor->getSizeInBytes());  // Add this line
outputTensor.second = std::move(tensor);

This has minimal performance impact since it runs only once per request and buffer sizes are typically small.

Alternatives

Currently, we manually zero the tensor after each inference on the Python side:

for output in response.additional_outputs:
    output.tensor.zero_()

This works but requires users to be aware of this behavior and implement the workaround themselves.

Additional context

This issue affects anyone using additionalOutput API for:

• Streaming inference with hidden_states
• Speech synthesis pipelines
• Any use case where partial buffer reads occur

Environment:

• TensorRT-LLM version: v0.18.0 branch
• Use case: Text-to-Speech with streaming

Thank you for the excellent work on TensorRT-LLM! We hope this feature can help other teams using additional outputs in streaming scenarios.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.