间断显卡编号的设备映射问题(English translation: device_mapping cannot handle discontinuous GPU indexes)
- Dominant language
- Python
- Stars
- 3.4k
- Forks
- 312
- Avg merge
- 1h 2m
- Merged PRs (30d)
- 2
Description
配置文件和资源调度似乎不支持映射到不从0开始或编号间断的显卡,例如`device_mapping: list([1, 3])`,运行时下列代码都会出现问题:
(English translation: Configs and resource manager seemingly fail to support discontinuous GPU indexes, such as `device_mapping: list([1, 3])`. Following codes cannot handle configs like that:)
```Python
# roll/configs/base_config.py: 257
if len(total_devices) > 0:
max_gpu_num = max(total_devices) + 1
if max_gpu_num <= self.num_gpus_per_node:
self.num_nodes = 1
else:
self.num_nodes = (max_gpu_num + self.num_gpus_per_node - 1) // self.num_gpus_per_node
```
```Python
# roll/distributed/scheduler/resource_manager.py
num_gpus_per_worker = len(device_mapping) // world_size
grouped_ranks = [
list(device_mapping[i : i + num_gpus_per_worker])
for i in range(0, len(device_mapping), num_gpus_per_worker)
]
for group in grouped_ranks:
pg_list = []
for rank in group:
node_rank = rank // self.gpu_per_node
gpu_rank = rank % self.gpu_per_node
assert node_rank < self.num_nodes, (f"device_mapping used gpus are more than "
f"num_nodes×num_gpus_per_node={self.num_nodes}×{self.gpu_per_node}")
pg = self.nodes_placement_group(node_rank)
pg_list.append(
dict(node_rank=node_rank, gpu_rank=gpu_rank, placement_group=pg, ray_address=ray_address)
)
allocated_pg.append(pg_list)
```
如何修改配置文件或程序代码来兼容间断的显卡编号?
(English translation: How to adapt configs or codes to discontinuous GPU indexes?)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the device-count logic in roll/configs/base_config.py around line 257 and the grouping and placement logic in roll/distributed/scheduler/resource_manager.py. Trace how device_mapping values such as [1, 3] determine node and GPU ranks, then verify that discontinuous indexes produce valid resource placement without invalid node calculations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100