huggingface / huggingface/candle

`slice_assign` is too slow

Open
#1,441 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

Hello,

The `slice_assign` function are too slow and it is causing a huge speed degradation in Rust version [ApplyTimestampRules](https://github.com/openai/whisper/blob/8bc8860694949db53c42ba47ddc23786c2e02a8b/whisper/decoding.py#L441).

Here is a quick comparison with the PyTorch implementation.

```rust
use candle_core::{Device, Tensor};
use quanta::Clock;

fn main() -> Result<(), Box> {
let clock = Clock::new();
let device = Device::cuda_if_available(0)?;
println!("Device: {:?}", &device);
let mut matrix = Tensor::new(vec![vec![0.0_f32; 10000]; 10000], &device)?;
let minus_inf = Tensor::new(vec![vec![f32::MIN; 10000]; 10000], &device)?;
let rows = matrix.dim(0)?;
let cols = matrix.dim(1)?;

let start = clock.now();
for _ in 0..1000 {
matrix = matrix.slice_assign(&[0..rows, 0..cols], &minus_inf)?;
}
let end = clock.now();
println!("1000 slice assign used {:?}", end - start);
Ok(())
}
```

```python3
from datetime import datetime

import torch

device = torch.device('cuda:0')
print("Device:", device)
matrix = torch.zeros(size=(10000, 10000)).to(device)
rows, cols = matrix.size()

start = datetime.now()
for _ in range(1000):
matrix[0 : rows, 0 : cols] = float('-inf')
end = datetime.now()

print("1000 slice assign used", (end - start).)
```

And the result:
```text
# cargo run --release --features=cuda --bin=rust_bench
Finished release [optimized] target(s) in 0.22s
Running `target/release/rust_bench`
Device: Cuda(CudaDevice(DeviceId(1)))
1000 slice assign used 1.865307185s

# python3 python_bench.py
Device: cuda:0
1000 slice assign used 0:00:00.014332
```

About 132 times slower :-(

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the Rust Tensor::slice_assign entry point and reproduce the supplied CUDA benchmark, comparing its 1000 full-matrix assignments with the PyTorch result. Done means the full-slice assignment no longer shows the reported roughly 132x slowdown.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.