huggingface / huggingface/candle

Unable to build candle with flash attention on iOS

Open
#3,015 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

When I try to build and run a llama 3.2 1b model on iOS (iPhone 14) with flash attention on Metal, I get ``/Users/jpchen/.cargo/git/checkouts/candle-6740f55d69a3bf41/b4ec636/candle-transformers/src/models/llama.rs:254:5:
not implemented: compile with '--features flash-attn'`

A little unfamiliar with Candle - I see that flash attention is supported for Metal hardware, and I was curious if this is an ios specific thing or if theres a way I could build it to get flash attention support? Thanks.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at candle-transformers/src/models/llama.rs around line 254 and trace how the flash-attn feature is enabled for Metal builds. Reproduce the iOS build with the llama 3.2 1b model, then determine whether flash attention is supported on iOS and what a supported build path requires. Done means the iOS behavior and required configuration are established.

Written by the indexing model from the issue text.

Assessment

Tech stack
ios, rust
Domain
build-system, machine-learning, mobile-dev
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.