open-compass / open-compass/VLMEvalKit
多进程推理，设置 --mode=infer 时，dist.barrier 的位置导致的阻塞问题
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.4k
- Forks
- 768
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 17
Description
您好！我最近在使用该仓库进行评测，发现在多卡运行并且设置 --mode 为 infer 时，在推理完第一个数据集之后会卡住。检查了一下 run.py 发现是因为 rank=0 的进程在 https://github.com/lihytotoro/VLMEvalKit/blob/main/run.py#L337 的地方会处理评测事宜，但由于只进行推理，因此会在 if 内部 continue 返回循环顶部，而其他的进程会跳过这个 if 到达下方 https://github.com/lihytotoro/VLMEvalKit/blob/main/run.py#L415 处的 dist.barrier，从而被阻塞，由于一直等不到 rank=0 到达因此会一直卡住，直到 nccl 超时退出。
我感觉这里的 dist.barrier 的位置应该可以有所调整，从而避免上述的现象。
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in run.py around lines 337 and 415, and reproduce multi-GPU evaluation with --mode=infer. Trace how rank 0 handles evaluation after the first dataset while other ranks reach dist.barrier. Done means inference continues across datasets without a process hanging or ending in an NCCL timeout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100