huggingface / huggingface/candle

Cross-CUDA Tensor::to_device fails with CUDA_ERROR_INVALID_CONTEXT (no peer-access enable)

Open
#3,524 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

## Bug

`Tensor::to_device(&Device::new_cuda(M))` from a tensor on `Device::new_cuda(N)` (where `N != M`) fails with `CUDA_ERROR_INVALID_CONTEXT` on first use.

## Reproduction

```rust
use candle_core::{Device, Tensor};

fn main() -> anyhow::Result<()> {
let dev0 = Device::new_cuda(0)?;
let dev1 = Device::new_cuda(1)?;
let t = Tensor::randn(0f32, 1., (1024, 1024), &dev0)?;
// ↓ errors with CUDA_ERROR_INVALID_CONTEXT
let _t_on_dev1 = t.to_device(&dev1)?;
Ok(())
}
```

Tested on 2× A6000 with NVLink (also reproduces over PCIe-only). cudarc 0.19.4, CUDA 13. The error path is deterministic — happens on the very first cross-card transfer in a fresh process.

## Root cause

`Tensor::to_device` → `CudaStorage::transfer_to_device` (`candle-core/src/cuda_backend/mod.rs:1261`) → `cudarc::CudaStream::clone_dtod`. cudarc's `memcpy_dtod` correctly detects that source and destination contexts differ and dispatches to `result::memcpy_peer_async`:

```rust
// cudarc-0.19.4 driver/safe/core.rs:1591
pub fn memcpy_dtod(...) -> Result<...> {
self.ctx.bind_to_thread()?;
let src_ctx = src.stream().context();
let dst_ctx = self.context();
...
if src_ctx == dst_ctx {
unsafe { result::memcpy_dtod_async(dst, src, num_bytes, self.cu_stream) }
} else {
unsafe {
result::memcpy_peer_async(
dst_ctx.cu_ctx, dst,
src_ctx.cu_ctx, src,
num_bytes, self.cu_stream,
)
}
}
}
```

`cuMemcpyPeerAsync` requires `cuCtxEnablePeerAccess` to have been called between the two contexts beforehand. Candle never calls it, so the driver rejects the very first cross-context transfer with `CUDA_ERROR_INVALID_CONTEXT`.

## Suggested fix

Two possible API shapes:

**(A) Explicit opt-in** — new public method:
```rust
pub fn enable_peer_access(&self, other: &Device) -> Result<()>
```
Idempotent (same-ordinal pairs no-op; `PEER_ACCESS_ALREADY_ENABLED` mapped to `Ok(())`); enables both directions in one call. Operators call it once between every pair they intend to do cross-card transfers across.

**(B) Auto-enable in `BackendDevice::new`** — opportunistically enable peer access between the new `CudaDevice` and all already-constructed `CudaDevice`s. Requires global state tracking in the cuda_backend. Larger surgery, surprising side effects (every `Device::new_cuda(N)` mutates global driver state), but zero caller burden.

I'd lean (A) for its explicitness — peer access has real costs (UVA address-space coordination, no-op on hardware that doesn't support it) and surfacing the opt-in keeps those costs visible. But (B) matches the principle of least surprise more closely; the call should "just work."

## Workaround until fixed

Host-staged transfer via CPU works at worst-case bandwidth:
```rust
let mid = t.to_device(&Device::Cpu)?;
let t_on_dev1 = mid.to_device(&dev1)?;
```

## Adjacent context

PR #1388 (Feb 2024, still open) noted that multi-GPU is "not well covered at the moment" per @LaurentMazare. This is one of the load-bearing primitives that needs to work for any multi-GPU model-parallel or unified-encoder/decoder pattern; happy to send a PR for whichever of (A) or (B) maintainers prefer.

I have a working implementation of option (A) on a fork branch and can open it as a PR right after this issue if (A) is acceptable.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at candle-core/src/cuda_backend/mod.rs:1261 and trace CudaStorage::transfer_to_device into cudarc's CudaStream::clone_dtod and peer-copy path. Reproduce the first cross-device transfer with two CUDA devices, then verify that the chosen peer-access behavior makes the transfer succeed without the CPU workaround and is covered by an appropriate regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
hpc, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.