huggingface / huggingface/candle

Tensor::arange allocates first on the CPU even when the target device is a GPU

Open
#1,078 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

`Tensor::arange` calls `Tensor::arange_step` which works by pushing values to a `Vec` stored on the CPU. I think this could be more efficient (firstly by precomputing the length and using `Vec::with_capacity`), and second to make the tensor on the device directly if it is supposed to end up on the GPU.

A parallel algorithm could be to use the thread index of the CUDA thread to compute the value `tensor[idx] = start + step * idx`. You could theoretically also do this on the CPU with something like rayon, but probably only for large tensors.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with Tensor::arange and its arange_step implementation, then trace how values are collected into the CPU Vec and transferred to the target device. Compare the proposed preallocated collection and CUDA thread-index approach, and consider the CPU path before verifying that arange produces the expected values directly on the requested device.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.