[Bug]: AutoDeploy model-coverage hits NVRTC compilation failed across multiple models
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
System Info
- CPU architecture: x86_64
- GPU properties: not captured from CI artifact
- Libraries
- TensorRT-LLM branch or tag: main (pipeline context)
- Container used: AutoDeploy model-coverage CI container
- OS: CI environment
- Additional context: terminal pipeline 52785950
Who can help?
No response
Information
- The official example scripts
- My own modified scripts
Tasks
- An officially supported task in the
examplesfolder (such as GLUE/SQuAD, ...) - My own task or dataset (give details below)
Reproduction
Pipeline 52785720 resolved to terminal model-coverage pipeline 52785950.
Affected registry entries currently disabled in examples/auto_deploy/model_registry/models.yaml include:
zai-org/GLM-5-FP8MiniMaxAI/MiniMax-M2.5MiniMaxAI/MiniMax-M2.7deepseek-ai/DeepSeek-V3.2deepseek-ai/DeepSeek-V3.2-SpecialeMiniMaxAI/MiniMax-M2deepseek-ai/DeepSeek-Prover-V2-671Bdeepseek-ai/DeepSeek-V3Qwen/Qwen3-0.6B-FP8
Representative evidence from per-test stdout logs ends with:
RuntimeError: NVRTC compilation failed
One concrete example is MiniMaxAI/MiniMax-M2.7, where the traceback tail runs through tensorrt_llm/_torch/auto_deploy/custom_ops/quantization/trtllm_quant.py into torch.ops.trtllm.fp8_block_scaling_gemm(...) before failing with RuntimeError: NVRTC compilation failed.
Expected behavior
These model-coverage entries should either compile and run successfully, or fail with a more specific unsupported-configuration signal if FP8 block-scaling GEMM is not supported in the current path.
actual behavior
TensorRT-LLM fails during AutoDeploy execution with RuntimeError: NVRTC compilation failed across multiple distinct models, suggesting a shared runtime or codegen failure rather than a model-specific problem.
additional notes
This looks like a TensorRT-LLM / AutoDeploy runtime issue, not a Hugging Face access problem. The same failure string appears across nine disabled entries, which makes it a good candidate for one general issue
Before submitting a new issue...
- Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with terminal pipeline 52785950 and the disabled entries in examples/auto_deploy/model_registry/models.yaml. Trace the MiniMaxAI/MiniMax-M2.7 failure through tensorrt_llm/_torch/auto_deploy/custom_ops/quantization/trtllm_quant.py and the fp8_block_scaling_gemm operation. Done means the affected entries run successfully or report a specific unsupported-configuration signal instead of the generic NVRTC compilation failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- backend, ci-cd, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100