huggingface / huggingface/candle

inefficient implementation of gelu for fp16

Open
#1,982 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

I'm running the dinov2 example on CPU on a Cortex-A76 computer, except I've quantised it to fp16. Looking at its perf profile, a large subset is due to running scalar numeric operations.

image

Tracking this down, I found that this was due to the gelu implementation:

```rust
#[inline(always)]
fn f16(v: f16) -> f16 {
f16::from_f32_const(0.5)
* v
* (f16::ONE
+ f16::tanh(
(f16::from_f32_const(2.0) / f16::PI).sqrt()
* v
* (f16::ONE + f16::from_f32_const(0.044715) * v * v),
))
}
```
It computes `sqrt(2/pi)` on every call, and over the totality of DinoV2, this represents 8.8% of cycles. I can't speak to whether this would be different for the f32 or f64 impls, but due to challenges of float math I cannot imagine that it is.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.