`nlp.pipe(n_process>1)` crashes with [E018]/[E871] for any pipeline using floret-mode vectors (e.g. sv_core_news_lg)
- Dominant language
- Python
- Stars
- 33.9k
- Forks
- 4.7k
- Avg merge
- 3m
- Merged PRs (30d)
- 1
Description
(🤖 I used Claude Code both to investigate the bug and write this issue)
## How to reproduce the behaviour
`nlp.pipe(texts, n_process=N)` with `N > 1` reliably raises `[E018] Can't retrieve string for hash '...'` (wrapped in `[E871] Error encountered in nlp.pipe with multiprocessing`) for any pipeline whose `vocab.vectors.mode == "floret"`. It does **not** happen for pipelines using `"default"` mode vectors.
This reproduces with a fully public, official spaCy pipeline — no custom training or private data involved:
```python
import spacy
# sv_core_news_lg uses floret-mode vectors (verified: nlp.vocab.vectors.mode == "floret").
nlp = spacy.load("sv_core_news_lg")
texts = [
"Regeringen presenterade i dag ett nytt forslag om klimatpolitik och energiproduktion.",
"Fotbollslaget vann matchen med tre mal mot noll efter en stark andra halvlek.",
"Forskare vid Uppsala universitet har publicerat en ny studie om havsnivahojningar.",
"Kommunen planerar att bygga fler bostader i centrala delar av staden nasta ar.",
"Det nya museet oppnar for allmanheten i september med en utstallning om vikingatiden.",
] * 200 # 1000 docs -- enough to fill several multiprocessing batches
if __name__ == "__main__":
docs = list(nlp.pipe(texts, batch_size=40, n_process=2))
print(f"OK: processed {len(docs)} docs without error")
```
Setup:
```bash
pip install spacy==3.8.7
python -m spacy download sv_core_news_lg
python repro.py
```
Result: crashes before completing a single document (well under a second of actual processing time — this is not a rare/late failure, it is immediate and deterministic).
Full traceback
```
Traceback (most recent call last):
File "repro.py", line 17, in
docs = list(nlp.pipe(texts, batch_size=40, n_process=2))
File ".../spacy/language.py", line 1622, in pipe
for doc in docs:
File ".../spacy/language.py", line 1688, in _multiprocessing_pipe
...
File ".../spacy/language.py", line 2410, in _apply_pipes
byte_docs = [(doc.to_bytes(), doc._context, None) for doc in docs]
File ".../spacy/util.py", line 1714, in _pipe
yield from proc.pipe(docs, **kwargs)
File "spacy/pipeline/transition_parser.pyx", line 245, in pipe
File ".../spacy/util.py", line 1661, in minibatch
batch = list(itertools.islice(items, int(batch_size)))
File ".../spacy/util.py", line 1714, in _pipe
yield from proc.pipe(docs, **kwargs)
[... repeated _pipe/minibatch frames through the chained pipeline components ...]
File "spacy/pipeline/trainable_pipe.pyx", line 75, in spacy.pipeline.trainable_pipe.TrainablePipe.pipe
File ".../spacy/pipeline/tok2vec.py", line 121, in predict
tokvecs = self.model.predict(docs)
File ".../thinc/model.py", line 334, in predict
return self._func(self, X, is_train=False)[0]
File ".../thinc/layers/chain.py", line 54, in forward
Y, inc_layer_grad = layer(X, is_train=is_train)
File ".../thinc/model.py", line 310, in __call__
return self._func(self, X, is_train=is_train)
File ".../thinc/layers/concatenate.py", line 57, in forward
Ys, callbacks = zip(*[layer(X, is_train=is_train) for layer in model.layers])
File ".../spacy/ml/staticvectors.py", line 55, in forward
V = vocab.vectors.get_batch(keys)
File "spacy/vectors.pyx", line 485, in spacy.vectors.Vectors.get_batch
File "spacy/strings.pyx", line 185, in spacy.strings.StringStore.as_string
File "spacy/strings.pyx", line 162, in spacy.strings.StringStore.__getitem__
KeyError: "[E018] Can't retrieve string for hash '90292280375180053'. This usually refers to an
issue with the `Vocab` or `StringStore`."
```
## Root cause (verified experimentally against the source, with an A/B fix test)
*(This section has been corrected since the issue was first filed: the original text blamed the `Doc.to_bytes()`/`from_bytes()` round trip. Further isolation showed that is not the mechanism — the docs never leave the worker before the crash. The actual cause is `Vocab` **pickling**, and a one-line fix has been verified against a source build of current `master`; see the PR.)*
The crash site is the floret string lookup. `spacy/ml/staticvectors.py::forward()` branches on `vocab.vectors.mode`:
```python
elif isinstance(vocab.vectors, Vectors) and vocab.vectors.mode == Mode.floret:
V = vocab.vectors.get_batch(keys)
```
`spacy/vectors.pyx::Vectors.get_batch()`, floret branch:
```python
elif self.mode == Mode.floret:
keys = [self.strings.as_string(key) for key in keys] # <- raises here
```
Floret vectors need each token's **literal text** (not just its hash) to slice character n-grams for the hashing trick, so they look the hash back up in `self.strings` — the `Vectors` object's own `StringStore` reference. When a pipeline is loaded normally, that reference is **the same object** as `vocab.strings` (`spacy.load` / `Vocab.__init__` establish this invariant), so every string the tokenizer interns is visible to the floret lookup.
Multiprocessing breaks that invariant. `Language._multiprocessing_pipe` hands each `mp.Process` bound methods (`self._ensure_doc_with_context` and each component's `proc.pipe`), so under the `spawn` start method (the default on macOS and Windows) the whole `Language`/`Vocab` gets **pickled** into the worker. And the `Vocab` pickle path severs the
sharing:
- `Vectors.__reduce__` (`spacy/vectors.pyx`) serializes via `to_bytes()`, and `unpickle_vectors` reconstructs the `Vectors` with its own **private, frozen copy** of the `StringStore` — no longer the object `vocab.strings` points to.
- `unpickle_vocab` (`spacy/vocab.pyx`) rewires `vocab.strings` and `vocab.vectors` but never re-links them.
In the worker, the tokenizer then interns each incoming token's text into `vocab.strings`, but the floret lookup consults the severed frozen copy in `vectors.strings` — so `as_string()` raises E018 for the first token whose string isn't in the model's stored string table. Immediate and deterministic, exactly as observed. `default`-mode vectors never call `as_string()` (they key
rows by hash directly), which is why they're immune.
Each link in this chain was verified in isolation (spaCy 3.8.7 and a source build of current `master`, 3.8.14):
- After `pickle.loads(pickle.dumps(nlp.vocab))`, `vocab.strings is vocab.vectors.strings` flips from `True` to `False`.
- Pickling exactly what the worker receives (the bound `_ensure_doc_with_context` + `proc.pipe` methods) and running the pipeline **in a single process** reproduces E018 with the *identical failing hash* as the multiprocessing run.
- Setting `vectors.strings = vocab.strings` on the unpickled objects makes the same pipeline run clean.
- Consistent with the pickle mechanism, the bug is **spawn-specific**:
`mp.set_start_method("fork")` (no pickling; Linux's default) runs the full repro to completion on the otherwise-failing setup. Linux users likely never see this, which would explain why it went unreported.
(The originally-suspected mechanism — `Doc.to_bytes()`/`from_bytes()` string loss as in #12407 — is ruled out: the traceback fires inside the worker *before* any result is serialized back, and the worker tokenizes raw text itself, so its `vocab.strings` demonstrably contains every needed string. Only the vectors' severed copy doesn't.)
## Fix (verified)
One line in `unpickle_vocab` (`spacy/vocab.pyx`), restoring the invariant that `Vocab.__init__` and deserialization already establish:
```diff
cdef Vocab vocab = Vocab()
vocab.vectors = vectors
vocab.strings = sstore
+ vocab.vectors.strings = vocab.strings
vocab.morphology = morphology
```
A/B-tested against a source build of `master` (3.8.14): unpatched build fails the repro above with the identical E018/hash; patched build processes all 1000 docs with `n_process=2`, and the `spacy/tests/vocab_vectors` + `spacy/tests/serialize` suites pass (222 passed).
**Workarounds until a fix lands** (both verified):
1. Force the fork start method before loading the model (POSIX only, with the usual fork-plus-threads caveats): `multiprocessing.set_start_method("fork")`.
2. Re-share the store on unpickle via `copyreg` — the module must be importable in the workers:
```python
# floret_mp_fix.py — import and call install() before nlp.pipe(n_process>1)
import copyreg
from spacy.vocab import Vocab, pickle_vocab, unpickle_vocab
def unpickle_vocab_fixed(*args):
vocab = unpickle_vocab(*args)
vocab.vectors.strings = vocab.strings
return vocab
def pickle_vocab_fixed(vocab):
_, args = pickle_vocab(vocab)
return unpickle_vocab_fixed, args
def install():
copyreg.pickle(Vocab, pickle_vocab_fixed)
```
Searched the tracker for prior reports of this specific combination (`gh search issues` across `explosion/spaCy` for `"floret" "multiprocess"`, `"StaticVectors multiprocessing"`, `"get_batch StringStore"`) and found nothing — filing this as it appears unreported, despite `default`-mode vectors clearly working fine under the same `n_process` codepath.
## Impact
Any pipeline trained with floret-mode vectors (a deliberately more compact vector representation, used by several of spaCy's own official non-English `_lg` pipelines, e.g. `sv_core_news_lg`) is silently incompatible with `nlp.pipe(n_process>1)` under the `spawn` multiprocessing start method (the default on macOS and Windows; Linux defaults to `fork` and is unaffected unless spawn is selected explicitly) for **any** component that reaches `StaticVectors` — which in practice means the parser and/or NER components in the standard `_core_news_lg` architectures, since they typically share a `tok2vec` that includes a
`StaticVectors` layer. The failure is immediate and 100% reproducible, not a rare race — so this should be very easy to confirm, but is also easy to miss until someone actually tries `n_process>1` with a floret pipeline (`default`-mode pipelines never hit it, which is presumably why it's gone unreported).
## Your Environment
* Operating System: macOS-15.5-arm64-arm-64bit
* Python Version Used: 3.10.11
* spaCy Version Used: 3.8.7
## Info about spaCy
- **spaCy version:** 3.8.7
- **Platform:** macOS-15.5-arm64-arm-64bit
- **Python version:** 3.10.11
- **Pipelines:** sv_core_news_lg (3.8.0)
Also reproduced the same failure signature against a privately-trained floret-vector pipeline (different language, same spaCy version) before narrowing the repro down to the fully public `sv_core_news_lg` case above — this is not specific to one model's training data.
Contributor guide
Research direction
Start by reading unpickle_vocab in spacy/vocab.pyx and the serialization tests under spacy/tests/vocab_vectors and spacy/tests/serialize. Reproduce the spawn failure with sv_core_news_lg, then add coverage for the pickled floret-vector pipeline. Done means n_process=2 completes successfully and the relevant test suites pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100