huggingface / huggingface/candle

Compare 0-dimension tensor got CUDA_ERROR_INVALID_VALUE

Open
#2,327 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

I got error CUDA_ERROR_INVALID_VALUE when running the code on cuda device. But the same code is running fine in cpu device.

```
use candle_core::{Device, DType, Tensor};

fn main() -> Result<(), Box> {
// let device = Device::Cpu;
let device = Device::new_cuda(0)?;

let a = Tensor::zeros((2, 1, 1, 0), DType::F32, &device)?;
let b = Tensor::zeros((2, 1, 1, 0), DType::F32, &device)?;

let diff = a.eq(&b)?;
println!("{diff}");
Ok(())
}
```

Error message:

Error: WithBacktrace { inner: Cuda(Cuda(DriverError(CUDA_ERROR_INVALID_VALUE, "invalid argument"))), backtrace: Backtrace [{ fn: "candle_core::error::Error::bt", file: "/root/.cargo/git/checkouts/candle-0c2b4fa9e5801351/a226a97/candle-core/src/error.rs", line: 231 }, { fn: " as candle_core::cuda_backend::error::WrapErr>::w::{{closure}}", file: "/root/.cargo/git/checkouts/candle-0c2b4fa9e5801351/a226a97/candle-core/src/cuda_backend/error.rs", line: 60 }, { fn: "core::result::Result::map_err", file: "/rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/result.rs", line: 829 }, { fn: " as candle_core::cuda_backend::error::WrapErr>::w", file: "/root/.cargo/git/checkouts/candle-0c2b4fa9e5801351/a226a97/candle-core/src/cuda_backend/error.rs", line: 60 }, { fn: "::f", file: "/root/.cargo/git/checkouts/candle-0c2b4fa9e5801351/a226a97/candle-core/src/cuda_backend/mod.rs", line: 968 }, { fn: "candle_core::cuda_backend::utils::Map2Any::map", file: "/root/.cargo/git/checkouts/candle-0c2b4fa9e5801351/a226a97/candle-core/src/cuda_backend/utils.rs", line: 166 }, { fn: "::cmp", file: "/root/.cargo/git/checkouts/candle-0c2b4fa9e5801351/a226a97/candle-core/src/cuda_backend/mod.rs", line: 1261 }, { fn: "candle_core::storage::Storage::cmp", file: "/root/.cargo/git/checkouts/candle-0c2b4fa9e5801351/a226a97/candle-core/src/storage.rs", line: 142 }, { fn: "candle_core::tensor::Tensor::cmp", file: "/root/.cargo/git/checkouts/candle-0c2b4fa9e5801351/a226a97/candle-core/src/tensor.rs", line: 991 }, { fn: "candle_core::tensor::Tensor::eq", file: "/root/.cargo/git/checkouts/candle-0c2b4fa9e5801351/a226a97/candle-core/src/tensor.rs", line: 1000 }, { fn: "myapp::main", file: "./src/main.rs", line: 10 }, { fn: "core::ops::function::FnOnce::call_once", file: "/rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/ops/function.rs", line: 250 }, { fn: "std::sys_common::backtrace::__rust_begin_short_backtrace", file: "/rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/sys_common/backtrace.rs", line: 155 }, { fn: "std::rt::lang_start::{{closure}}", file: "/rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/rt.rs", line: 159 }, { fn: "core::ops::function::impls:: for &F>::call_once", file: "/rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/ops/function.rs", line: 284 }, { fn: "std::panicking::try::do_call", file: "/rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/panicking.rs", line: 559 }, { fn: "std::panicking::try", file: "/rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/panicking.rs", line: 523 }, { fn: "std::panic::catch_unwind", file: "/rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/panic.rs", line: 149 }, { fn: "std::rt::lang_start_internal::{{closure}}", file: "/rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/rt.rs", line: 141 }, { fn: "std::panicking::try::do_call", file: "/rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/panicking.rs", line: 559 }, { fn: "std::panicking::try", file: "/rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/panicking.rs", line: 523 }, { fn: "std::panic::catch_unwind", file: "/rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/panic.rs", line: 149 }, { fn: "std::rt::lang_start_internal", file: "/rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/rt.rs", line: 141 }, { fn: "std::rt::lang_start", file: "/rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/rt.rs", line: 158 }, { fn: "main" }, { fn: "__libc_start_main" }, { fn: "_start" }] }

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the CUDA failure with the example in the issue, then inspect Tensor::eq in candle_core/src/tensor.rs and the CUDA comparison path in candle_core/src/cuda_backend/mod.rs, including utils.rs. Confirm that comparing the two zero-sized tensors succeeds on CUDA as it does on CPU, and add or update coverage where the existing comparison behavior is exercised.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.