[Feature]: Add offloading support to visual_gen to reduce GPU memory usage

🚀 The feature, motivation and pitch

Feature request

I would like to add offloading support to tensorrt_llm/_torch/visual_gen.

The goal is to reduce peak GPU memory usage by allowing selected parts of visual generation pipelines, such as model components, auxiliary modules, or guardrails, to be offloaded when they are not actively needed during generation.

Motivation

This would help support more memory-heavy visual generation workloads on a wider range of GPUs.

In particular, it would be useful for:

• GPUs with limited VRAM, including consumer/workstation GPUs
• larger visual generation models
• longer video generation
• multi-view generation
• pipelines that include extra components such as safety or guardrail models

Initial experiments suggest that offloading can significantly reduce peak GPU memory usage with limited latency overhead.

Proposal

Add a general offloading mechanism to visual_gen that can be configured by users and reused across visual generation models, rather than relying on model-specific pipeline changes.

I would be interested in contributing an initial implementation and would appreciate guidance on the preferred design/API for making this upstreamable.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.