[Feature]: LoRA support for FP4 (NVFP4) quantization

🚀 The feature, motivation and pitch

Problem: In tensorrt_llm/quantization/layers.py, FP4Linear and FP4RowLinear assert lora_runtime_params is None with "lora is not supported on FP4Linear now", so NVFP4 engines cannot load LoRA adapters at runtime while FP8 can.

Motivation: We need to serve NVFP4 base models with per-request LoRA (e.g. task-specific). FP4 + LoRA in one engine would match the FP8 workflow.

Pitch: Add LoRA support for FP4 the same way as FP8: remove the assertion, keep FP16/BF16 activations for the LoRA path, run LoRA after the FP4 GEMM and add the result to the output (for row-linear, before allreduce). Handle FP8 inputs (e.g. FP8 KV cache) by dequantizing before LoRA.

We have a local patch that does this for the single-GPU, non–gemm_allreduce_plugin path and have verified inference. It does not cover the fused GEMM+allreduce path or other plugin/config combinations, so we are asking for proper upstream support that covers all paths rather than relying on our patch. Happy to share our changes or contribute to an upstream implementation.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.