Checkpointing is not compatible with .grad(), please use .backward() if possible

when training deepseek v3 model using cuda_graph, it reports the following error:
Checkpointing is not compatible with .grad(), please use .backward() if possible

setting the following args:
--recompute-granularity selective
--recompute-modules mla_up_proj mlp

--cuda-graph-impl transformer_engine
--cuda-graph-scope full

Reproduce DeepSeek V3 training with CUDA graphs using selective recomputation for mla_up_proj and mlp, transformer_engine, and full scope. Start by tracing the checkpointing path that rejects .grad() and identify the interaction with the CUDA-graph configuration; done means the reported configuration works or its incompatibility is clearly documented.

Written by the indexing model from the issue text.