alibaba / alibaba/ROLL

间断显卡编号的设备映射问题(English translation: device_mapping cannot handle discontinuous GPU indexes)

Open
#205 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.4k
Forks
312
Avg merge
1h 2m
Merged PRs (30d)
2

Description

配置文件和资源调度似乎不支持映射到不从0开始或编号间断的显卡,例如`device_mapping: list([1, 3])`,运行时下列代码都会出现问题:
(English translation: Configs and resource manager seemingly fail to support discontinuous GPU indexes, such as `device_mapping: list([1, 3])`. Following codes cannot handle configs like that:)

```Python
# roll/configs/base_config.py: 257
if len(total_devices) > 0:
max_gpu_num = max(total_devices) + 1
if max_gpu_num <= self.num_gpus_per_node:
self.num_nodes = 1
else:
self.num_nodes = (max_gpu_num + self.num_gpus_per_node - 1) // self.num_gpus_per_node
```
```Python
# roll/distributed/scheduler/resource_manager.py
num_gpus_per_worker = len(device_mapping) // world_size
grouped_ranks = [
list(device_mapping[i : i + num_gpus_per_worker])
for i in range(0, len(device_mapping), num_gpus_per_worker)
]
for group in grouped_ranks:
pg_list = []
for rank in group:
node_rank = rank // self.gpu_per_node
gpu_rank = rank % self.gpu_per_node

assert node_rank < self.num_nodes, (f"device_mapping used gpus are more than "
f"num_nodes×num_gpus_per_node={self.num_nodes}×{self.gpu_per_node}")

pg = self.nodes_placement_group(node_rank)
pg_list.append(
dict(node_rank=node_rank, gpu_rank=gpu_rank, placement_group=pg, ray_address=ray_address)
)
allocated_pg.append(pg_list)
```
如何修改配置文件或程序代码来兼容间断的显卡编号?
(English translation: How to adapt configs or codes to discontinuous GPU indexes?)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the device-count logic in roll/configs/base_config.py around line 257 and the grouping and placement logic in roll/distributed/scheduler/resource_manager.py. Trace how device_mapping values such as [1, 3] determine node and GPU ranks, then verify that discontinuous indexes produce valid resource placement without invalid node calculations.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.