huggingface / huggingface/candle
How to run inference of a (very) large model across mulitple GPUs ?
Open
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
It is mentioned on README that candle supports multi GPU inference, using NCCL under the hood. How can this be implemented ? I wonder if there is any available example to look at..
Also, I know PyTorch has things like DDP and FSDP, is candle support for multi GPU inference comparable to these techniques ?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.