en_core_web_hftrf-3.8.1 fails to load with spacy-transformers 1.4.0 (safetensors weights)
- Dominant language
- Python
- Stars
- 33.9k
- Forks
- 4.7k
- Avg merge
- 3m
- Merged PRs (30d)
- 1
Description
## Summary
`en_core_web_hftrf-3.8.1` ([release](https://github.com/explosion/spacy-models/releases/tag/en_core_web_hftrf-3.8.1)) ships with transformer weights in safetensors format and pins `spacy-transformers>=1.4.0,<1.5.0`. However, `spacy-transformers` 1.4.0 cannot load safetensors weights — `HFShim.from_bytes` calls `torch.load()` which only handles pickle/zip format.
The safetensors migration ([explosion/spacy-transformers#423](https://github.com/explosion/spacy-transformers/pull/423), commit 2cba136) landed on master 3 days after the 1.4.0 release, so there is currently no released version of `spacy-transformers` that can load this model.
## Steps to reproduce
```bash
pip install spacy==3.8.7 spacy-transformers==1.4.0
pip install en_core_web_hftrf==3.8.1
python -c "import spacy; spacy.load('en_core_web_hftrf')"
```
## Error
```
_pickle.UnpicklingError: Weights only load failed. [...]
WeightsUnpickler error: Unsupported operand 64
```
With `weights_only=False` forced, the error becomes:
```
_pickle.UnpicklingError: invalid load key, '@'
```
The first byte of the state is `0x40` (`@`), which is the start of a safetensors header (little-endian u64 header length), not a pickle stream.
## Environment
- Python 3.12
- spacy 3.8.7
- spacy-transformers 1.4.0
- torch 2.8.0
## Request
Could a new release of `spacy-transformers` be cut to include the safetensors migration from explosion/spacy-transformers#423? Currently there is no way to use `en_core_web_hftrf-3.8.1` with a released version of `spacy-transformers`.
Contributor guide
Assessment
This issue has not been assessed yet.