[Bug] train error

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/SpecForge/discussions/new/choose Otherwise, it will be closed.
• 5. Please use English, otherwise it will be closed.

Describe the bug

The memory usage isn’t that high, but it still throws an OutOfMemoryError. What’s the issue?

error：
[rank3]: self._make_launchers() [rank3]: File "/opt/miniconda/envs/sglang/lib/python3.12/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 573, in _make_launchers [rank3]: raise RuntimeError(f"No valid triton configs. {type(exc).name}: {exc}") [rank3]: torch._inductor.exc.InductorError: RuntimeError: No valid triton configs. OutOfMemoryError: out of resource: triton_tem_fused_0 Required: 107008 Hardware limit:101376 Reducing block sizes or num_stages may help.

Reproduction

torchrun
--standalone
--nproc_per_node 8
scripts/train_eagle3_online.py
--target-model-path /models/Qwen3-14B
--train-data-path dataset/opc_train_100000.jsonl
--output-dir outputs/Qwen3-14B-eagle3
--num-epochs 10
--batch-size 1
--learning-rate 1e-4
--max-length 2048
--chat-template qwen
--cache-dir cache
--tp-size 8
2>&1 | tee train.log

Environment

8X L20