10b的模型如何实现单实例单机多卡推理
Open
Nobody has claimed this yet.
enhancement
- Dominant language
- Python
- Stars
- 2.4k
- Forks
- 176
- PR merge metrics
- No merged PRs in 30d
Description
示例代码是将模型加载在单张显卡上推理的，如何将模型权重分配在多张卡上，有像hugging face那样简单的device_map='auto'的方法吗？
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the example code referenced in the issue and trace how the model is loaded and inference is initialized. Check whether the current entry point supports distributing weights across multiple GPUs or whether this requires a new design. Done should mean that the supported multi-GPU approach is implemented or clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100