sgl-project / sgl-project/SpecForge
[Bug] invalid device ordinal while using multi node in data prepare
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 347
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 41
Description
Checklist
- 1. I have searched related issues but cannot get the expected help.
- 2. The bug has not been fixed in the latest version.
- 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/SpecForge/discussions/new/choose Otherwise, it will be closed.
- 5. Please use English, otherwise it will be closed.
Describe the bug
When I prepare offline training dataset, OOM happended, so I try to use 2 nodes to apply TP=16 for the target model. But I got a RuntimeError:
[rank11]: File "/mnt/data/ceyu.cy/SpecForge/scripts/prepare_hidden_states.py", line 88, in __init__
[rank11]: self.model_runner, _ = load_model(self.server_args, self.port_args, tp_rank)
[rank11]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank11]: File "/root/sglang/python/sglang/bench_one_batch.py", line 144, in load_model
[rank11]: model_runner = ModelRunner(
[rank11]: ^^^^^^^^^^^^
[rank11]: File "/root/sglang/python/sglang/srt/model_executor/model_runner.py", line 235, in __init__
[rank11]: min_per_gpu_memory = self.init_torch_distributed()
[rank11]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank11]: File "/root/sglang/python/sglang/srt/model_executor/model_runner.py", line 507, in init_torch_distributed
[rank11]: torch.get_device_module(self.device).set_device(self.gpu_id)
[rank11]: File "/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py", line 529, in set_device
[rank11]: torch._C._cuda_setDevice(device)
[rank11]: RuntimeError: CUDA error: invalid device ordinal
[rank11]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
[rank11]: For debugging consider passing CUDA_LAUNCH_BLOCKING=1
[rank11]: Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
[rank13]: Traceback (most recent call last):
[rank13]: File "/mnt/data/ceyu.cy/SpecForge/scripts/prepare_hidden_states.py", line 383, in <module>
[rank13]: main()
[rank13]: File "/mnt/data/ceyu.cy/SpecForge/scripts/prepare_hidden_states.py", line 376, in main
[rank13]: hidden_states_generator = SglangHiddenStatesGenerator(
[rank13]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank13]: File "/mnt/data/ceyu.cy/SpecForge/scripts/prepare_hidden_states.py", line 88, in __init__
[rank13]: self.model_runner, _ = load_model(self.server_args, self.port_args, tp_rank)
[rank13]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank13]: File "/root/sglang/python/sglang/bench_one_batch.py", line 144, in load_model
[rank13]: model_runner = ModelRunner(
[rank13]: ^^^^^^^^^^^^
[rank13]: File "/root/sglang/python/sglang/srt/model_executor/model_runner.py", line 235, in __init__
[rank13]: min_per_gpu_memory = self.init_torch_distributed()
[rank13]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank13]: File "/root/sglang/python/sglang/srt/model_executor/model_runner.py", line 507, in init_torch_distributed
[rank13]: torch.get_device_module(self.device).set_device(self.gpu_id)
[rank13]: File "/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py", line 529, in set_device
[rank13]: torch._C._cuda_setDevice(device)
[rank13]: RuntimeError: CUDA error: invalid device ordinal
[rank13]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
[rank13]: For debugging consider passing CUDA_LAUNCH_BLOCKING=1
[rank13]: Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Reproduction
torchrun --nproc_per_node=8 --nnodes=2 \
scripts/prepare_hidden_states.py \
...
**--tp-size 16** \
...
Environment
Name: specforge
Version: 0.1.0
Summary: SpecForge Project
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with scripts/prepare_hidden_states.py, especially SglangHiddenStatesGenerator and its call to load_model, then inspect sglang/bench_one_batch.py and ModelRunner.init_torch_distributed. Reproduce with torchrun --nproc_per_node=8 --nnodes=2 and --tp-size 16; done means the multi-node data-preparation run no longer raises invalid device ordinal.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100