huggingface / huggingface/candle
Tensor::arange allocates first on the CPU even when the target device is a GPU
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
`Tensor::arange` calls `Tensor::arange_step` which works by pushing values to a `Vec` stored on the CPU. I think this could be more efficient (firstly by precomputing the length and using `Vec::with_capacity`), and second to make the tensor on the device directly if it is supposed to end up on the GPU.
A parallel algorithm could be to use the thread index of the CUDA thread to compute the value `tensor[idx] = start + step * idx`. You could theoretically also do this on the CPU with something like rayon, but probably only for large tensors.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with Tensor::arange and its arange_step implementation, then trace how values are collected into the CPU Vec and transferred to the target device. Compare the proposed preallocated collection and CUDA thread-index approach, and consider the CPU path before verifying that arange produces the expected values directly on the requested device.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- machine-learning, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100