huggingface / huggingface/candle

Running models with different precisions

Open
#2,032 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

I am testing different model architectures, and when loading the model weights (e.g. for falcon or mamba architectures) with precision either `bf16` or `f16` I usually get this error:

`Candle error: 'unexpected dtype, expected: F32, got: BF16'`

I am running the examples on candle, and passing in precision of `f16` or `bf16`. Is there a way around this by tweaking the code ? Or should I load weights directly from `f16/bf16` precision through some other repo in HuggingFace ?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Candle examples that load Falcon or Mamba model weights and trace how the `f16` and `bf16` precision options reach weight loading. Reproduce the reported `expected: F32, got: BF16` error, then determine whether the precision options and weight dtypes can be handled consistently. Done means the affected examples load successfully with the requested precision.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.