huggingface / huggingface/candle

Added support for Vulkan, any interest?

Open
#3,272 0 comments 21 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

I have a Intel Arc A770 16GB GPU and wanted to use it with candle.
I took niklasha's work on niklas-vulkan-2 branch cherry-pick's into the current main branch.
I (when I say I, I mean I was the navigator, Codex 5.2 max did the work) added the following:

Added Vulkan queue-family selection and synchronize() so VulkanDevice uses compute-capable queues and can block on GPU work (device.rs).
Expanded Vulkan storage surface with raw_buffer() access for kernel dispatch and fixed error wiring (storage.rs).
Wired Vulkan kernel registry to include matmul, norms, softmax, masked softmax, and quantized kernels (lib.rs).
Added F32/F16 matmul shader stubs and norm/softmax shaders for initial Vulkan ops coverage (*.comp).
Implemented Vulkan masked softmax and staged SDPA path with GQA support in candle-nn (ops.rs).
Added Vulkan smoke tests and masked softmax correctness test (vulkan_smoke_tests.rs, vulkan_masked_softmax.rs).
Fixed missing imports and push-constant binding for Vulkan command execution (storage.rs).
Added bytemuck + vulkano-shaders feature wiring for Vulkan builds (Cargo.toml).
Introduced QVulkanStorage backed by raw byte buffers with dequantize/quantize helpers (vulkan.rs).
Added Vulkan quantized matmul kernels for Q5_0 and Q8_0 (naive, F32 output) (qmatmul_q5_0_f32.comp, qmatmul_q8_0_f32.comp).
Hooked Vulkan quantized path into QTensor forward and added Vulkan quantized tests (mod.rs, vulkan_quantized_tests.rs).
Added a dequantize‑fallback backward path for QLoRA-style gradients (mod.rs).
Cleaned up dummy Vulkan stubs to match new quantized API surface (dummy_vulkan.rs).
Fixed multiple test harness macro/feature mismatches to compile with Vulkan enabled (test_utils.rs, *.rs).

Contributor guide

No contributing guide indexed for this repository

Research direction

Review the Vulkan changes listed across device.rs, storage.rs, lib.rs, ops.rs, vulkan.rs, mod.rs, and the named shader and test files to understand the proposed scope. Run the Vulkan smoke, masked-softmax, and quantized tests with Vulkan enabled, then determine whether this broad implementation is wanted and what maintainers consider complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.