huggingface / huggingface/candle
`slice_assign` is too slow
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
Hello,
The `slice_assign` function are too slow and it is causing a huge speed degradation in Rust version [ApplyTimestampRules](https://github.com/openai/whisper/blob/8bc8860694949db53c42ba47ddc23786c2e02a8b/whisper/decoding.py#L441).
Here is a quick comparison with the PyTorch implementation.
```rust
use candle_core::{Device, Tensor};
use quanta::Clock;
fn main() -> Result<(), Box> {
let clock = Clock::new();
let device = Device::cuda_if_available(0)?;
println!("Device: {:?}", &device);
let mut matrix = Tensor::new(vec![vec![0.0_f32; 10000]; 10000], &device)?;
let minus_inf = Tensor::new(vec![vec![f32::MIN; 10000]; 10000], &device)?;
let rows = matrix.dim(0)?;
let cols = matrix.dim(1)?;
let start = clock.now();
for _ in 0..1000 {
matrix = matrix.slice_assign(&[0..rows, 0..cols], &minus_inf)?;
}
let end = clock.now();
println!("1000 slice assign used {:?}", end - start);
Ok(())
}
```
```python3
from datetime import datetime
import torch
device = torch.device('cuda:0')
print("Device:", device)
matrix = torch.zeros(size=(10000, 10000)).to(device)
rows, cols = matrix.size()
start = datetime.now()
for _ in range(1000):
matrix[0 : rows, 0 : cols] = float('-inf')
end = datetime.now()
print("1000 slice assign used", (end - start).)
```
And the result:
```text
# cargo run --release --features=cuda --bin=rust_bench
Finished release [optimized] target(s) in 0.22s
Running `target/release/rust_bench`
Device: Cuda(CudaDevice(DeviceId(1)))
1000 slice assign used 1.865307185s
# python3 python_bench.py
Device: cuda:0
1000 slice assign used 0:00:00.014332
```
About 132 times slower :-(
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the Rust Tensor::slice_assign entry point and reproduce the supplied CUDA benchmark, comparing its 1000 full-matrix assignments with the PyTorch result. Done means the full-slice assignment no longer shows the reported roughly 132x slowdown.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100