NER resizing error in combination with entity rulers
- Dominant language
- Python
- Stars
- 33.9k
- Forks
- 4.7k
- Avg merge
- 3m
- Merged PRs (30d)
- 1
Description
There's most likely a bug in spaCy v2 around the NER resizing, entity rulers in the pipeline, and IO+(re)initialization.
Background in this discussion thread: https://github.com/explosion/spaCy/discussions/8864. There are several entity rulers and NER models in the pipeline. At a certain point, when the entity ruler is deserialized from disk and calls `nlp.pipe` when adding its patterns internally, the NER tries to resize itself and gets stuck in an inconsistent state with incompatible network sizes:
```
(...)
File "C:\Users\smisr\PycharmProjects\june_2021\virtualenv\lib\site-packages\spacy\util.py", line 690, in from_disk
reader(path / key)
File "C:\Users\smisr\PycharmProjects\june_2021\virtualenv\lib\site-packages\spacy\pipeline\entityruler.py", line 344, in
"patterns": lambda p: self.add_patterns(
File "C:\Users\smisr\PycharmProjects\june_2021\virtualenv\lib\site-packages\spacy\pipeline\entityruler.py", line 226, in add_patterns
for label, pattern, ent_id in zip(
File "C:\Users\smisr\PycharmProjects\june_2021\virtualenv\lib\site-packages\spacy\language.py", line 829, in pipe
for doc in docs:
(...)
File "nn_parser.pyx", line 274, in spacy.syntax.nn_parser.Parser.predict
(...)
File "_parser_model.pyx", line 269, in spacy.syntax._parser_model.ParserModel.resize_output
ValueError: could not broadcast input array from shape (1258,64) into shape (1226,64)
```
I'm not sure if this behaviour also occurs in spaCy v3. It feels like a niche case though, and for now there is a usable work-around (add all entity ruler labels to the NER up-front). Logging here for future reference in case we happen to stumble upon similar problems.
I don't think this is very high priority, but if want to debug & fix this, we'll need a reproducible code snippet first.
Contributor guide
Assessment
This issue has not been assessed yet.