huggingface / huggingface/candle

[Cuda] u32 tensor div by f64 leads to zeroed out tensor

Open
#3,177 9 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

# Description
## Summary
Okay, so looks like I stumbled upon an obscure bug here. I tried reducing the bug to as small as an MRP as possible, but it looks like its cause is a number of issues that simply results in my code being the perfect storm. What I do know is that the `to_device()` call at line 92 is where the backtrace is pointing towards, and that the program requires a few loops to run before the crash occurs (downstream data corruption, perhaps?).

## Environment
- OS: Windows 11 x64
- candle-core: 0.9.1
- GPU: RTX 4060
- CUDA toolkit: 12.4

# Backtrace
```
C:\Users\[USER]\.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\candle-kernels-0.9.1\src\indexing.cu:134: block: [0,0,0], thread: [0,0,0] Assertion `idx < dst_dim_size` failed.
C:\Users\[USER]\.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\candle-kernels-0.9.1\src\indexing.cu:134: block: [0,0,0], thread: [1,0,0] Assertion `idx < dst_dim_size` failed.
C:\Users\[USER]\.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\candle-kernels-0.9.1\src\indexing.cu:134: block: [0,0,0], thread: [2,0,0] Assertion `idx < dst_dim_size` failed.
Error: DriverError(CUDA_ERROR_ASSERT, "device-side assert triggered")
0: std::backtrace_rs::backtrace::win64::trace
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library\std\src\..\..\backtrace\src\backtrace\win64.rs:85
1: std::backtrace_rs::backtrace::trace_unsynchronized
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library\std\src\..\..\backtrace\src\backtrace\mod.rs:66
2: std::backtrace::Backtrace::create
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library\std\src\backtrace.rs:331
3: std::backtrace::Backtrace::capture
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library\std\src\backtrace.rs:296
4: enum2$::bt
at C:\Users\[USER]\.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\candle-core-0.9.1\src\error.rs:255
5: candle_core::cuda_backend::error::impl$1::w::closure$0,cudarc::driver::result::DriverError>
at C:\Users\[USER]\.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\candle-core-0.9.1\src\cuda_backend\error.rs:60
6: enum2$,cudarc::driver::result::DriverError> >::map_err,cudarc::driver::result::DriverError,enum2$,candle_core::cuda_backend:
at C:\Users\[USER]\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\result.rs:856
7: candle_core::cuda_backend::error::impl$1::w,cudarc::driver::result::DriverError>
at C:\Users\[USER]\.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\candle-core-0.9.1\src\cuda_backend\error.rs:60
8: candle_core::cuda_backend::impl$29::to_cpu_storage
at C:\Users\[USER]\.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\candle-core-0.9.1\src\cuda_backend\mod.rs:1526
9: candle_core::tensor::Tensor::to_device
at C:\Users\[USER]\.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\candle-core-0.9.1\src\tensor.rs:2130
10: to_device_bug::topk_event_indices
at .\src\main.rs:91
11: to_device_bug::SensorVideo::poll
at .\src\main.rs:59
12: to_device_bug::main
at .\src\main.rs:185
13: core::ops::function::FnOnce::call_once,enum2$ > > (*)(),tuple$<> >
at C:\Users\[USER]\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\ops\function.rs:250
14: std::sys::backtrace::__rust_begin_short_backtrace,enum2$ > > (*)(),enum2$,enum2$ > > >
at C:\Users\[USER]\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\std\src\sys\backtrace.rs:152
15: std::rt::lang_start::closure$0,enum2$ > > >
at C:\Users\[USER]\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\std\src\rt.rs:199
16: std::rt::lang_start_internal::closure$0
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library\std\src\rt.rs:168
17: std::panicking::try::do_call
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library\std\src\panicking.rs:589
18: std::panicking::try
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library\std\src\panicking.rs:552
19: std::panic::catch_unwind
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library\std\src\panic.rs:359
20: std::rt::lang_start_internal
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library\std\src\rt.rs:164
21: std::rt::lang_start,enum2$ > > >
at C:\Users\[USER]\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\std\src\rt.rs:198
22: main
23: invoke_main
at D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78
24: __scrt_common_main_seh
at D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
25: BaseThreadInitThunk
26: RtlUserThreadStart

error: process didn't exit successfully: `target\debug\to_device_bug.exe` (exit code: 1)
```

# MRP Download
[mrp.zip](https://github.com/user-attachments/files/23452501/mrp.zip)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running the attached mrp.zip on the reported Windows, RTX 4060, and CUDA 12.4 setup, then trace src/main.rs at lines 91, 59, and 185. Compare the u32 tensor division by f64 with the indexing.cu assertion at line 134 and determine where the tensor becomes invalid. Done means the reproduction no longer produces a zeroed tensor or device-side assertion.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.