huggingface / huggingface/candle
Cross-CUDA Tensor::to_device fails with CUDA_ERROR_INVALID_CONTEXT (no peer-access enable)
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
## Bug
`Tensor::to_device(&Device::new_cuda(M))` from a tensor on `Device::new_cuda(N)` (where `N != M`) fails with `CUDA_ERROR_INVALID_CONTEXT` on first use.
## Reproduction
```rust
use candle_core::{Device, Tensor};
fn main() -> anyhow::Result<()> {
let dev0 = Device::new_cuda(0)?;
let dev1 = Device::new_cuda(1)?;
let t = Tensor::randn(0f32, 1., (1024, 1024), &dev0)?;
// ↓ errors with CUDA_ERROR_INVALID_CONTEXT
let _t_on_dev1 = t.to_device(&dev1)?;
Ok(())
}
```
Tested on 2× A6000 with NVLink (also reproduces over PCIe-only). cudarc 0.19.4, CUDA 13. The error path is deterministic — happens on the very first cross-card transfer in a fresh process.
## Root cause
`Tensor::to_device` → `CudaStorage::transfer_to_device` (`candle-core/src/cuda_backend/mod.rs:1261`) → `cudarc::CudaStream::clone_dtod`. cudarc's `memcpy_dtod` correctly detects that source and destination contexts differ and dispatches to `result::memcpy_peer_async`:
```rust
// cudarc-0.19.4 driver/safe/core.rs:1591
pub fn memcpy_dtod(...) -> Result<...> {
self.ctx.bind_to_thread()?;
let src_ctx = src.stream().context();
let dst_ctx = self.context();
...
if src_ctx == dst_ctx {
unsafe { result::memcpy_dtod_async(dst, src, num_bytes, self.cu_stream) }
} else {
unsafe {
result::memcpy_peer_async(
dst_ctx.cu_ctx, dst,
src_ctx.cu_ctx, src,
num_bytes, self.cu_stream,
)
}
}
}
```
`cuMemcpyPeerAsync` requires `cuCtxEnablePeerAccess` to have been called between the two contexts beforehand. Candle never calls it, so the driver rejects the very first cross-context transfer with `CUDA_ERROR_INVALID_CONTEXT`.
## Suggested fix
Two possible API shapes:
**(A) Explicit opt-in** — new public method:
```rust
pub fn enable_peer_access(&self, other: &Device) -> Result<()>
```
Idempotent (same-ordinal pairs no-op; `PEER_ACCESS_ALREADY_ENABLED` mapped to `Ok(())`); enables both directions in one call. Operators call it once between every pair they intend to do cross-card transfers across.
**(B) Auto-enable in `BackendDevice::new`** — opportunistically enable peer access between the new `CudaDevice` and all already-constructed `CudaDevice`s. Requires global state tracking in the cuda_backend. Larger surgery, surprising side effects (every `Device::new_cuda(N)` mutates global driver state), but zero caller burden.
I'd lean (A) for its explicitness — peer access has real costs (UVA address-space coordination, no-op on hardware that doesn't support it) and surfacing the opt-in keeps those costs visible. But (B) matches the principle of least surprise more closely; the call should "just work."
## Workaround until fixed
Host-staged transfer via CPU works at worst-case bandwidth:
```rust
let mid = t.to_device(&Device::Cpu)?;
let t_on_dev1 = mid.to_device(&dev1)?;
```
## Adjacent context
PR #1388 (Feb 2024, still open) noted that multi-GPU is "not well covered at the moment" per @LaurentMazare. This is one of the load-bearing primitives that needs to work for any multi-GPU model-parallel or unified-encoder/decoder pattern; happy to send a PR for whichever of (A) or (B) maintainers prefer.
I have a working implementation of option (A) on a fork branch and can open it as a PR right after this issue if (A) is acceptable.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at candle-core/src/cuda_backend/mod.rs:1261 and trace CudaStorage::transfer_to_device into cudarc's CudaStream::clone_dtod and peer-copy path. Reproduce the first cross-device transfer with two CUDA devices, then verify that the chosen peer-access behavior makes the transfer succeed without the CPU workaround and is covered by an appropriate regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- hpc, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100