huggingface / huggingface/candle

`get_or_load_func` drops the kernel name from CUDA symbol-lookup errors

Open Beginner friendly
#3,884 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

## Summary

When a CUDA kernel symbol is missing from a loaded module, the resulting error
does not say which kernel. `CudaDevice::get_or_load_func` takes the name as an
argument and discards it on the failure path:

```rust
// candle-core/src/cuda_backend/device.rs
pub fn get_or_load_func(&self, fn_name: &str, mdl: &kernels::Module) -> Result {
let ms = self.modules.read().unwrap();
if let Some(mdl) = ms.mdls[mdl.index()].as_ref() {
let func = mdl.load_function(fn_name).w()?; // <- fn_name dropped
...
}
drop(ms);
let mut ms = self.modules.write().unwrap();
let cuda_module = self.context.load_module(mdl.ptx().into()).w()?;
ms.mdls[mdl.index()] = Some(cuda_module.clone());
let func = cuda_module.load_function(fn_name).w()?; // <- and here
...
}
```

`.w()` wraps cudarc's `DriverError` unchanged, so the caller gets:

```
DriverError(CUDA_ERROR_NOT_FOUND, "named symbol not found")
```

The name that would make this actionable was in scope at the point of failure
and was thrown away.

## Why this is worse than it looks

Two things conspire to make it hard to recover the information afterwards.

**A backtrace does not identify the kernel.** The error propagates upward as a
value through `?`; the eventual panic happens at whatever `unwrap`/`expect`
consumes it, often several layers up in user code. `RUST_BACKTRACE=1` names
that consumer, not the `get_or_load_func` call site. The natural first
debugging step yields nothing.

**A missing symbol is a legitimate, reachable state, not a corrupt build.**
`candle-kernels` guards kernels by target architecture, and the guards are not
uniform across a dtype. For `F8E4M3` specifically, the casts sit under one
threshold and the elementwise and indexing ops under a higher one:

| kernels | guard |
|---|---|
| `cast_f8_e4m3_f32`, `cast_f8_e4m3_bf16`, … (`cast.cu`) | `__CUDA_ARCH__ >= 800` |
| `affine_f8_e4m3` (`affine.cu`) | `__CUDA_ARCH__ >= 890` |
| `is_*_f8_e4m3`, `gather_*_f8_e4m3`, `scatter_add_f8` (`indexing.cu`) | `__CUDA_ARCH__ >= 890` |

So a build for, say, `sm_86` has a genuinely partial `F8E4M3` kernel set: some
operations on such a tensor work and others fail at lookup time. That is a
reasonable design — but it means "symbol not found" is a *routine* outcome of
running a supported dtype on a supported device, and the one thing the user
needs in order to understand it is the name that is currently discarded.

Concretely: we hit `CUDA_ERROR_NOT_FOUND` while building a model on an
`sm_86` device and, after reading the guards above and tracing the load path by
hand, still cannot say which kernel is missing. That is the entire content of
this report — not that the lookup failed, but that a failed lookup is
undiagnosable from its own error.

## Why not fix this in cudarc

`DriverError` is a newtype over the raw driver result:

```rust
pub struct DriverError(pub sys::CUresult);
```

It has a public field, and its `std::error::Error` impl is behind
`#[cfg(feature = "std")]`. Carrying a symbol name would require an owned
`String` — allocation in a `no_std`-friendly FFI binding, a change to a
pattern-matchable public shape, and a breaking change across every API that
returns it. cudarc already surfaces the driver's own description; the caller's
identifier is the caller's to attach.

## Shape of the fix

Attach the name at both `load_function` sites, and the module with it, since
"which module was searched" is the natural follow-up question:

```
cuda: kernel `affine_f8_e4m3` not found in module `affine`
(this kernel may not be compiled for the target architecture)
```

The parenthetical matters as much as the name: for a user on a pre-`sm_89`
device, "not found" reads as a broken installation, when it is in fact the
architecture guard doing its job.

## Acceptance

- A failed lookup names the kernel and the module it was searched in.
- The underlying `DriverError` remains in the source chain, so
`CUDA_ERROR_NOT_FOUND` is still distinguishable from other driver failures.
- Both call sites are covered — the cached-module path and the
load-then-lookup path fail identically today and must keep doing so.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in candle-core/src/cuda_backend/device.rs at CudaDevice::get_or_load_func and inspect both load_function call sites, including the cached-module and load-then-lookup paths. Exercise or add coverage for a missing CUDA symbol and verify that both errors identify the kernel and module while preserving the underlying DriverError and CUDA_ERROR_NOT_FOUND.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.