[Bug][AutoDeploy]: Investigate accuracy drop in Super + MTP + CudaGraph

System Info

• 8 x H100
• In tests/integration/defs/accuracy/test_llm_api_autodeploy.py::TestNemotronSuperV3::test_mtp, we have noticed that the accuracy regularly fluctuates by about ~1% across runs, which has not been observed for Llama + Eagle or Super + MTP with torch-simple.

Resolution:

• Understand why accuracy fluctuates with torch-cudagraph and fix the underlying issue.

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

On 8xH100s, Run

tests/integration/defs/accuracy/test_llm_api_autodeploy.py::TestNemotronSuperV3::test_mtp[8-trtllm]

the accuracy often hovers around 91.5-92%. If we change torch-cudagraph to torch-simple, this goes to consistently around 92.5%.

Expected behavior

Run

tests/integration/defs/accuracy/test_llm_api_autodeploy.py::TestNemotronSuperV3::test_mtp[8-trtllm]

The accuracy should be the same regardless of whether we use torch-cudagraph or torch-simple, and should be close to the 92.7% reference accuracy that we use for the base Super model (no MTP).

actual behavior

Run

tests/integration/defs/accuracy/test_llm_api_autodeploy.py::TestNemotronSuperV3::test_mtp[8-trtllm]

The accuracy hovers around 91.5% with torch-cudagraph (current behavior) and around 92.5% with torch-simple (modified in test for debugging).

additional notes

none

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.