explosion / explosion/spaCy

en_core_web_hftrf-3.8.1 fails to load with spacy-transformers 1.4.0 (safetensors weights)

Open
#13,958 2 comments 4 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
33.9k
Forks
4.7k
Avg merge
3m
Merged PRs (30d)
1

Description

## Summary

`en_core_web_hftrf-3.8.1` ([release](https://github.com/explosion/spacy-models/releases/tag/en_core_web_hftrf-3.8.1)) ships with transformer weights in safetensors format and pins `spacy-transformers>=1.4.0,<1.5.0`. However, `spacy-transformers` 1.4.0 cannot load safetensors weights — `HFShim.from_bytes` calls `torch.load()` which only handles pickle/zip format.

The safetensors migration ([explosion/spacy-transformers#423](https://github.com/explosion/spacy-transformers/pull/423), commit 2cba136) landed on master 3 days after the 1.4.0 release, so there is currently no released version of `spacy-transformers` that can load this model.

## Steps to reproduce

```bash
pip install spacy==3.8.7 spacy-transformers==1.4.0
pip install en_core_web_hftrf==3.8.1
python -c "import spacy; spacy.load('en_core_web_hftrf')"
```

## Error

```
_pickle.UnpicklingError: Weights only load failed. [...]
WeightsUnpickler error: Unsupported operand 64
```

With `weights_only=False` forced, the error becomes:

```
_pickle.UnpicklingError: invalid load key, '@'
```

The first byte of the state is `0x40` (`@`), which is the start of a safetensors header (little-endian u64 header length), not a pickle stream.

## Environment

- Python 3.12
- spacy 3.8.7
- spacy-transformers 1.4.0
- torch 2.8.0

## Request

Could a new release of `spacy-transformers` be cut to include the safetensors migration from explosion/spacy-transformers#423? Currently there is no way to use `en_core_web_hftrf-3.8.1` with a released version of `spacy-transformers`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.