huggingface / huggingface/candle

Copy to GPU Too Slow

Open
#1,699 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

I noticed that when loading a model onto the GPU, it copies to the CPU first and then onto the GPU. Is there a way that I can load the model directly onto GPU?
```
pub(crate) fn storage(&self, array: A) -> Result {
match self {
Device::Cpu => Ok(Storage::Cpu(array.to_cpu_storage())),
Device::Cuda(device) => {
let storage = array.to_cpu_storage();
let storage = device.storage_from_cpu_storage(&storage)?;
Ok(Storage::Cuda(storage))
}
Device::Metal(device) => {
let storage = array.to_cpu_storage();
let storage = device.storage_from_cpu_storage(&storage)?;
Ok(Storage::Metal(storage))
}
}
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the shown Device::storage entry point, including to_cpu_storage and storage_from_cpu_storage for CUDA and Metal. Determine how model loading can avoid the CPU intermediate for GPU targets, then verify that GPU loading preserves the existing Storage result and behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.