huggingface / huggingface/candle

Vulkan accelerator backend (Device::Vulkan): implementation available, seeking to upstream

Open
#3,985 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

## Summary

Proposing a Vulkan backend for candle-core analogous to the existing CUDA and Metal backends. A working reference implementation exists in the `rexlunae/joshua` fork. This issue is to gauge interest and agree a shape before upstreaming a ~2k-line backend.

## What the implementation provides

- `Device::Vulkan` / `DeviceLocation::Vulkan` + `VulkanDevice` / `VulkanStorage`, wired through the `Device`/`Storage`/`DeviceLocation` enums and all allocation/sync paths — mirrors the Metal/OpenCL-style backend structure.
- Uses **ash** with its `loaded` feature (libvulkan.so dlopen'd at runtime, no hard link) and **naga** to compile embedded GLSL compute shaders to SPIR-V at runtime (no external glslc).
- Bring-up target is unified-memory iGPUs (AMD Renoir / RADV): host-visible+coherent buffers make the host<->device round-trip a plain memcpy; native kernels cover matmul (register-blocked, handles transposed/batched/broadcast rhs), elementwise, and last-dim reduction.
- `CustomOp1`/`CustomOp2::vulkan_fwd` device-native hooks so softmax/RMSNorm-style CustomOps can run on-device instead of the CPU default (default remains a correct CPU round-trip).

## Validation (AMD Radeon / RADV RENOIR iGPU)

Native kernels are bit-exact vs CPU (matmul, elementwise, last-dim reduction). Device-native softmax matches CPU to ~7.5e-9 and RMSNorm to ~1e-7. Full 150B-MoE inference runs on-device via `--device vulkan`.

## Questions

1. Is there interest in a Vulkan backend upstream? It parallels the Metal backend's role (broader hardware coverage than CUDA-only), especially for iGPUs and RADV-based systems.
2. Shape preference: full backend in one PR, or a staged series (device/storage round-trip first, then kernels, then CustomOp hooks)?
3. Any concerns about the ash/naga dependencies (currently optional feature-gated behind `vulkan`)?

I'm happy to prepare the PR in the agreed shape using the joshua implementation as the reference.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the existing CUDA and Metal backends, then compare the working implementation in the rexlunae/joshua fork. Clarify whether the project wants one full PR or staged work covering device/storage round-trip, kernels, and CustomOp hooks. Done means an agreed upstream shape and a Vulkan backend validated against the stated CPU and RADV results.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.