huggingface / huggingface/candle
Vulkan accelerator backend (Device::Vulkan): implementation available, seeking to upstream
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
## Summary
Proposing a Vulkan backend for candle-core analogous to the existing CUDA and Metal backends. A working reference implementation exists in the `rexlunae/joshua` fork. This issue is to gauge interest and agree a shape before upstreaming a ~2k-line backend.
## What the implementation provides
- `Device::Vulkan` / `DeviceLocation::Vulkan` + `VulkanDevice` / `VulkanStorage`, wired through the `Device`/`Storage`/`DeviceLocation` enums and all allocation/sync paths — mirrors the Metal/OpenCL-style backend structure.
- Uses **ash** with its `loaded` feature (libvulkan.so dlopen'd at runtime, no hard link) and **naga** to compile embedded GLSL compute shaders to SPIR-V at runtime (no external glslc).
- Bring-up target is unified-memory iGPUs (AMD Renoir / RADV): host-visible+coherent buffers make the host<->device round-trip a plain memcpy; native kernels cover matmul (register-blocked, handles transposed/batched/broadcast rhs), elementwise, and last-dim reduction.
- `CustomOp1`/`CustomOp2::vulkan_fwd` device-native hooks so softmax/RMSNorm-style CustomOps can run on-device instead of the CPU default (default remains a correct CPU round-trip).
## Validation (AMD Radeon / RADV RENOIR iGPU)
Native kernels are bit-exact vs CPU (matmul, elementwise, last-dim reduction). Device-native softmax matches CPU to ~7.5e-9 and RMSNorm to ~1e-7. Full 150B-MoE inference runs on-device via `--device vulkan`.
## Questions
1. Is there interest in a Vulkan backend upstream? It parallels the Metal backend's role (broader hardware coverage than CUDA-only), especially for iGPUs and RADV-based systems.
2. Shape preference: full backend in one PR, or a staged series (device/storage round-trip first, then kernels, then CustomOp hooks)?
3. Any concerns about the ash/naga dependencies (currently optional feature-gated behind `vulkan`)?
I'm happy to prepare the PR in the agreed shape using the joshua implementation as the reference.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the existing CUDA and Metal backends, then compare the working implementation in the rexlunae/joshua fork. Clarify whether the project wants one full PR or staged work covering device/storage round-trip, kernels, and CustomOp hooks. Done means an agreed upstream shape and a Vulkan backend validated against the stated CPU and RADV results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100