huggingface / huggingface/candle
Copy to GPU Too Slow
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
I noticed that when loading a model onto the GPU, it copies to the CPU first and then onto the GPU. Is there a way that I can load the model directly onto GPU?
```
pub(crate) fn storage(&self, array: A) -> Result {
match self {
Device::Cpu => Ok(Storage::Cpu(array.to_cpu_storage())),
Device::Cuda(device) => {
let storage = array.to_cpu_storage();
let storage = device.storage_from_cpu_storage(&storage)?;
Ok(Storage::Cuda(storage))
}
Device::Metal(device) => {
let storage = array.to_cpu_storage();
let storage = device.storage_from_cpu_storage(&storage)?;
Ok(Storage::Metal(storage))
}
}
}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the shown Device::storage entry point, including to_cpu_storage and storage_from_cpu_storage for CUDA and Metal. Determine how model loading can avoid the CPU intermediate for GPU targets, then verify that GPU loading preserves the existing Storage result and behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100