huggingface / huggingface/candle
inefficient implementation of gelu for fp16
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
I'm running the dinov2 example on CPU on a Cortex-A76 computer, except I've quantised it to fp16. Looking at its perf profile, a large subset is due to running scalar numeric operations.
Tracking this down, I found that this was due to the gelu implementation:
```rust
#[inline(always)]
fn f16(v: f16) -> f16 {
f16::from_f32_const(0.5)
* v
* (f16::ONE
+ f16::tanh(
(f16::from_f32_const(2.0) / f16::PI).sqrt()
* v
* (f16::ONE + f16::from_f32_const(0.044715) * v * v),
))
}
```
It computes `sqrt(2/pi)` on every call, and over the totality of DinoV2, this represents 8.8% of cycles. I can't speak to whether this would be different for the f32 or f64 impls, but due to challenges of float math I cannot imagine that it is.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.