huggingface / huggingface/candle
Unable to build candle with flash attention on iOS
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
When I try to build and run a llama 3.2 1b model on iOS (iPhone 14) with flash attention on Metal, I get ``/Users/jpchen/.cargo/git/checkouts/candle-6740f55d69a3bf41/b4ec636/candle-transformers/src/models/llama.rs:254:5:
not implemented: compile with '--features flash-attn'`
A little unfamiliar with Candle - I see that flash attention is supported for Metal hardware, and I was curious if this is an ios specific thing or if theres a way I could build it to get flash attention support? Thanks.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at candle-transformers/src/models/llama.rs around line 254 and trace how the flash-attn feature is enabled for Metal builds. Reproduce the iOS build with the llama 3.2 1b model, then determine whether flash attention is supported on iOS and what a supported build path requires. Done means the iOS behavior and required configuration are established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ios, rust
- Domain
- build-system, machine-learning, mobile-dev
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100