huggingface / huggingface/candle

Error: DriverError(CUDA_ERROR_OUT_OF_MEMORY, "out of memory") with multiple GPU

Open
#2,046 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

I have: 4x RTX 3080 = 40GB total memory (each GPU by 10 GB memory)

I try to load model Mistral 7 about 15Gb file.

But I take error:

```
root@C.10529376:~/ai-server$ cargo run
Finished dev [unoptimized + debuginfo] target(s) in 0.20s
Running `target/debug/ai-server`
retrieved the files in 27.070873ms
Error: DriverError(CUDA_ERROR_OUT_OF_MEMORY, "out of memory")
```

Is it possible to run on multiple GPU mode?

Contributor guide

No contributing guide indexed for this repository

Research direction

The report names only cargo run and a Mistral 7 model; no source file or test is identified. Start by locating the model-loading path and GPU-device handling, then reproduce the CUDA out-of-memory failure and establish whether multi-GPU execution is supported or what implementation scope would be required.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
ai, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.