huggingface / huggingface/candle

Manually perform 8-bit dot product (__dp4a)

Open
#2,348 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

Hi,
The candle-kernel crate throws an error on the Maxwell architecture when migrating code that works fine in PyTorch.
It seems the only problem is '__dp4a' which used for performing four 8-bit integer dot product operations for quantization.
It would be great if the kernel crate has implementation to compute the dot product for '__CUDA_ARCH__ < 610'.

```cuda
#if __CUDA_ARCH__ < 610
// Manually perform 8-bit dot product
__device__ int manual_dp4a(int a, int b, int c) {
int result = c;
for (int i = 0; i < 4; ++i) {
int8_t a_byte = (a >> (i * 8)) & 0xFF;
int8_t b_byte = (b >> (i * 8)) & 0xFF;
result += a_byte * b_byte;
}
return result;
}
#endif

#if __CUDA_ARCH__ >= 610
...
sumi = __dp4a(vi0, u[2*i+0], sumi);
...
#else
...
sumi = manual_dp4a(vi0, u[2*i+0], sumi);
...
#endif
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in the candle-kernel crate and locate the __dp4a call sites and their __CUDA_ARCH__ guards. Compare the existing path for newer architectures with the manual_dp4a example in this issue, then verify that Maxwell builds and computes the same quantized dot products without __dp4a.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.