huggingface / huggingface/candle

BERT Safetensors variable mismatch

Open
#1,887 2 comments 2 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

Hi, I was running the BERT example code and noticed that some of the variables weren't correctly aligning with the current Safetensors obtained via:
```
let repo: ApiRepo = api.model("bert-base-uncased".to_string());
let weights_path: PathBuf = repo.get("model.safetensors")?
```

For example the model spec in [`candle-transformers/src/models/bert.rs`](https://github.com/huggingface/candle/blob/main/candle-transformers/src/models/bert.rs) results in: `Error: TensorNotFound("embeddings.word_embeddings.weight")`.

The Safetensors version prepends all variables with `bert` and uses the older `gamma`/`beta` notation. This issue has also been noted [here](https://github.com/naver/sqlova/issues/1).

I think the problem is in `layer_norm` which doesn't expect gamma and beta but weight and bias:
```
let weight = vb.get_with_hints(size, "weight", crate::Init::Const(1.))?;
let bias = if config.affine {
Some(vb.get_with_hints(size, "bias", crate::Init::Const(0.))?)
} else {
None
};
```

The Safetensor variables are as follows:
```
bert.embeddings.LayerNorm.beta
bert.embeddings.LayerNorm.gamma
bert.embeddings.position_embeddings.weight
bert.embeddings.token_type_embeddings.weight
bert.embeddings.word_embeddings.weight
bert.encoder.layer.0.attention.output.LayerNorm.beta
bert.encoder.layer.0.attention.output.LayerNorm.gamma
bert.encoder.layer.0.attention.output.dense.bias
bert.encoder.layer.0.attention.output.dense.weight
bert.encoder.layer.0.attention.self.key.bias
bert.encoder.layer.0.attention.self.key.weight
bert.encoder.layer.0.attention.self.query.bias
bert.encoder.layer.0.attention.self.query.weight
bert.encoder.layer.0.attention.self.value.bias
bert.encoder.layer.0.attention.self.value.weight
bert.encoder.layer.0.intermediate.dense.bias
bert.encoder.layer.0.intermediate.dense.weight
bert.encoder.layer.0.output.LayerNorm.beta
bert.encoder.layer.0.output.LayerNorm.gamma
bert.encoder.layer.0.output.dense.bias
bert.encoder.layer.0.output.dense.weight
bert.encoder.layer.1.attention.output.LayerNorm.beta
bert.encoder.layer.1.attention.output.LayerNorm.gamma
bert.encoder.layer.1.attention.output.dense.bias
bert.encoder.layer.1.attention.output.dense.weight
bert.encoder.layer.1.attention.self.key.bias
bert.encoder.layer.1.attention.self.key.weight
bert.encoder.layer.1.attention.self.query.bias
bert.encoder.layer.1.attention.self.query.weight
bert.encoder.layer.1.attention.self.value.bias
bert.encoder.layer.1.attention.self.value.weight
bert.encoder.layer.1.intermediate.dense.bias
bert.encoder.layer.1.intermediate.dense.weight
bert.encoder.layer.1.output.LayerNorm.beta
bert.encoder.layer.1.output.LayerNorm.gamma
bert.encoder.layer.1.output.dense.bias
bert.encoder.layer.1.output.dense.weight
bert.encoder.layer.10.attention.output.LayerNorm.beta
bert.encoder.layer.10.attention.output.LayerNorm.gamma
bert.encoder.layer.10.attention.output.dense.bias
bert.encoder.layer.10.attention.output.dense.weight
bert.encoder.layer.10.attention.self.key.bias
bert.encoder.layer.10.attention.self.key.weight
bert.encoder.layer.10.attention.self.query.bias
bert.encoder.layer.10.attention.self.query.weight
bert.encoder.layer.10.attention.self.value.bias
bert.encoder.layer.10.attention.self.value.weight
bert.encoder.layer.10.intermediate.dense.bias
bert.encoder.layer.10.intermediate.dense.weight
bert.encoder.layer.10.output.LayerNorm.beta
bert.encoder.layer.10.output.LayerNorm.gamma
bert.encoder.layer.10.output.dense.bias
bert.encoder.layer.10.output.dense.weight
bert.encoder.layer.11.attention.output.LayerNorm.beta
bert.encoder.layer.11.attention.output.LayerNorm.gamma
bert.encoder.layer.11.attention.output.dense.bias
bert.encoder.layer.11.attention.output.dense.weight
bert.encoder.layer.11.attention.self.key.bias
bert.encoder.layer.11.attention.self.key.weight
bert.encoder.layer.11.attention.self.query.bias
bert.encoder.layer.11.attention.self.query.weight
bert.encoder.layer.11.attention.self.value.bias
bert.encoder.layer.11.attention.self.value.weight
bert.encoder.layer.11.intermediate.dense.bias
bert.encoder.layer.11.intermediate.dense.weight
bert.encoder.layer.11.output.LayerNorm.beta
bert.encoder.layer.11.output.LayerNorm.gamma
bert.encoder.layer.11.output.dense.bias
bert.encoder.layer.11.output.dense.weight
bert.encoder.layer.2.attention.output.LayerNorm.beta
bert.encoder.layer.2.attention.output.LayerNorm.gamma
bert.encoder.layer.2.attention.output.dense.bias
bert.encoder.layer.2.attention.output.dense.weight
bert.encoder.layer.2.attention.self.key.bias
bert.encoder.layer.2.attention.self.key.weight
bert.encoder.layer.2.attention.self.query.bias
bert.encoder.layer.2.attention.self.query.weight
bert.encoder.layer.2.attention.self.value.bias
bert.encoder.layer.2.attention.self.value.weight
bert.encoder.layer.2.intermediate.dense.bias
bert.encoder.layer.2.intermediate.dense.weight
bert.encoder.layer.2.output.LayerNorm.beta
bert.encoder.layer.2.output.LayerNorm.gamma
bert.encoder.layer.2.output.dense.bias
bert.encoder.layer.2.output.dense.weight
bert.encoder.layer.3.attention.output.LayerNorm.beta
bert.encoder.layer.3.attention.output.LayerNorm.gamma
bert.encoder.layer.3.attention.output.dense.bias
bert.encoder.layer.3.attention.output.dense.weight
bert.encoder.layer.3.attention.self.key.bias
bert.encoder.layer.3.attention.self.key.weight
bert.encoder.layer.3.attention.self.query.bias
bert.encoder.layer.3.attention.self.query.weight
bert.encoder.layer.3.attention.self.value.bias
bert.encoder.layer.3.attention.self.value.weight
bert.encoder.layer.3.intermediate.dense.bias
bert.encoder.layer.3.intermediate.dense.weight
bert.encoder.layer.3.output.LayerNorm.beta
bert.encoder.layer.3.output.LayerNorm.gamma
bert.encoder.layer.3.output.dense.bias
bert.encoder.layer.3.output.dense.weight
bert.encoder.layer.4.attention.output.LayerNorm.beta
bert.encoder.layer.4.attention.output.LayerNorm.gamma
bert.encoder.layer.4.attention.output.dense.bias
bert.encoder.layer.4.attention.output.dense.weight
bert.encoder.layer.4.attention.self.key.bias
bert.encoder.layer.4.attention.self.key.weight
bert.encoder.layer.4.attention.self.query.bias
bert.encoder.layer.4.attention.self.query.weight
bert.encoder.layer.4.attention.self.value.bias
bert.encoder.layer.4.attention.self.value.weight
bert.encoder.layer.4.intermediate.dense.bias
bert.encoder.layer.4.intermediate.dense.weight
bert.encoder.layer.4.output.LayerNorm.beta
bert.encoder.layer.4.output.LayerNorm.gamma
bert.encoder.layer.4.output.dense.bias
bert.encoder.layer.4.output.dense.weight
bert.encoder.layer.5.attention.output.LayerNorm.beta
bert.encoder.layer.5.attention.output.LayerNorm.gamma
bert.encoder.layer.5.attention.output.dense.bias
bert.encoder.layer.5.attention.output.dense.weight
bert.encoder.layer.5.attention.self.key.bias
bert.encoder.layer.5.attention.self.key.weight
bert.encoder.layer.5.attention.self.query.bias
bert.encoder.layer.5.attention.self.query.weight
bert.encoder.layer.5.attention.self.value.bias
bert.encoder.layer.5.attention.self.value.weight
bert.encoder.layer.5.intermediate.dense.bias
bert.encoder.layer.5.intermediate.dense.weight
bert.encoder.layer.5.output.LayerNorm.beta
bert.encoder.layer.5.output.LayerNorm.gamma
bert.encoder.layer.5.output.dense.bias
bert.encoder.layer.5.output.dense.weight
bert.encoder.layer.6.attention.output.LayerNorm.beta
bert.encoder.layer.6.attention.output.LayerNorm.gamma
bert.encoder.layer.6.attention.output.dense.bias
bert.encoder.layer.6.attention.output.dense.weight
bert.encoder.layer.6.attention.self.key.bias
bert.encoder.layer.6.attention.self.key.weight
bert.encoder.layer.6.attention.self.query.bias
bert.encoder.layer.6.attention.self.query.weight
bert.encoder.layer.6.attention.self.value.bias
bert.encoder.layer.6.attention.self.value.weight
bert.encoder.layer.6.intermediate.dense.bias
bert.encoder.layer.6.intermediate.dense.weight
bert.encoder.layer.6.output.LayerNorm.beta
bert.encoder.layer.6.output.LayerNorm.gamma
bert.encoder.layer.6.output.dense.bias
bert.encoder.layer.6.output.dense.weight
bert.encoder.layer.7.attention.output.LayerNorm.beta
bert.encoder.layer.7.attention.output.LayerNorm.gamma
bert.encoder.layer.7.attention.output.dense.bias
bert.encoder.layer.7.attention.output.dense.weight
bert.encoder.layer.7.attention.self.key.bias
bert.encoder.layer.7.attention.self.key.weight
bert.encoder.layer.7.attention.self.query.bias
bert.encoder.layer.7.attention.self.query.weight
bert.encoder.layer.7.attention.self.value.bias
bert.encoder.layer.7.attention.self.value.weight
bert.encoder.layer.7.intermediate.dense.bias
bert.encoder.layer.7.intermediate.dense.weight
bert.encoder.layer.7.output.LayerNorm.beta
bert.encoder.layer.7.output.LayerNorm.gamma
bert.encoder.layer.7.output.dense.bias
bert.encoder.layer.7.output.dense.weight
bert.encoder.layer.8.attention.output.LayerNorm.beta
bert.encoder.layer.8.attention.output.LayerNorm.gamma
bert.encoder.layer.8.attention.output.dense.bias
bert.encoder.layer.8.attention.output.dense.weight
bert.encoder.layer.8.attention.self.key.bias
bert.encoder.layer.8.attention.self.key.weight
bert.encoder.layer.8.attention.self.query.bias
bert.encoder.layer.8.attention.self.query.weight
bert.encoder.layer.8.attention.self.value.bias
bert.encoder.layer.8.attention.self.value.weight
bert.encoder.layer.8.intermediate.dense.bias
bert.encoder.layer.8.intermediate.dense.weight
bert.encoder.layer.8.output.LayerNorm.beta
bert.encoder.layer.8.output.LayerNorm.gamma
bert.encoder.layer.8.output.dense.bias
bert.encoder.layer.8.output.dense.weight
bert.encoder.layer.9.attention.output.LayerNorm.beta
bert.encoder.layer.9.attention.output.LayerNorm.gamma
bert.encoder.layer.9.attention.output.dense.bias
bert.encoder.layer.9.attention.output.dense.weight
bert.encoder.layer.9.attention.self.key.bias
bert.encoder.layer.9.attention.self.key.weight
bert.encoder.layer.9.attention.self.query.bias
bert.encoder.layer.9.attention.self.query.weight
bert.encoder.layer.9.attention.self.value.bias
bert.encoder.layer.9.attention.self.value.weight
bert.encoder.layer.9.intermediate.dense.bias
bert.encoder.layer.9.intermediate.dense.weight
bert.encoder.layer.9.output.LayerNorm.beta
bert.encoder.layer.9.output.LayerNorm.gamma
bert.encoder.layer.9.output.dense.bias
bert.encoder.layer.9.output.dense.weight
bert.pooler.dense.bias
bert.pooler.dense.weight
cls.predictions.bias
cls.predictions.transform.LayerNorm.beta
cls.predictions.transform.LayerNorm.gamma
cls.predictions.transform.dense.bias
cls.predictions.transform.dense.weight
cls.seq_relationship.bias
cls.seq_relationship.weight
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in candle-transformers/src/models/bert.rs and inspect the BERT variable-binding paths, including layer_norm. Reproduce the BERT example with bert-base-uncased and its model.safetensors file, then verify that loading completes without TensorNotFound errors and that the listed bert-prefixed gamma/beta variables are handled.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.