huggingface / huggingface/candle
Error: DriverError(CUDA_ERROR_OUT_OF_MEMORY, "out of memory") with multiple GPU
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
I have: 4x RTX 3080 = 40GB total memory (each GPU by 10 GB memory)
I try to load model Mistral 7 about 15Gb file.
But I take error:
```
root@C.10529376:~/ai-server$ cargo run
Finished dev [unoptimized + debuginfo] target(s) in 0.20s
Running `target/debug/ai-server`
retrieved the files in 27.070873ms
Error: DriverError(CUDA_ERROR_OUT_OF_MEMORY, "out of memory")
```
Is it possible to run on multiple GPU mode?
Contributor guide
No contributing guide indexed for this repository
Research direction
The report names only cargo run and a Mistral 7 model; no source file or test is identified. Start by locating the model-loading path and GPU-device handling, then reproduce the CUDA out-of-memory failure and establish whether multi-GPU execution is supported or what implementation scope would be required.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- ai, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100