huggingface / huggingface/candle
Running models with different precisions
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
I am testing different model architectures, and when loading the model weights (e.g. for falcon or mamba architectures) with precision either `bf16` or `f16` I usually get this error:
`Candle error: 'unexpected dtype, expected: F32, got: BF16'`
I am running the examples on candle, and passing in precision of `f16` or `bf16`. Is there a way around this by tweaking the code ? Or should I load weights directly from `f16/bf16` precision through some other repo in HuggingFace ?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the Candle examples that load Falcon or Mamba model weights and trace how the `f16` and `bf16` precision options reach weight loading. Reproduce the reported `expected: F32, got: BF16` error, then determine whether the precision options and weight dtypes can be handled consistently. Done means the affected examples load successfully with the requested precision.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100