explosion / explosion/spaCy

EntityRecognizer throws IndexError when used in pipeline with Transformer and custom span getter

Open
#9,719 2 comments 0 reactions 0 assignees View on GitHub
feat / ner feat / transformer
Dominant language
Python
Stars
33.9k
Forks
4.7k
Avg merge
3m
Merged PRs (30d)
1

Description

EntityRecognizer throws IndexError when used in pipeline with Transformer and custom span getter during training:
```
File "/home/---/---/research_spacy_ru/.venv/lib/python3.8/site-packages/spacy/language.py", line 1122, in update
proc.update(examples, sgd=None, losses=losses, **component_cfg[name])
File "spacy/pipeline/transition_parser.pyx", line 416, in spacy.pipeline.transition_parser.Parser.update
File "spacy/ml/parser_model.pyx", line 293, in spacy.ml.parser_model.ParserStepModel.finish_steps
File "spacy/ml/parser_model.pyx", line 456, in spacy.ml.parser_model.precompute_hiddens.begin_update.backward
File "/home/---/---/research_spacy_ru/.venv/lib/python3.8/site-packages/spacy/ml/_precomputable_affine.py", line 49, in backward
Xf = X[ids]
IndexError: index 221 is out of bounds for axis 0 with size 221
```

## How to reproduce the behaviour
I created my custom span_getter: https://gist.github.com/tomateit/06e53b108f764e7240ea7ae8e2e830fd
It adapts number of words to respective number of word pieces, to better fit into transformer window.
Pipeline works with this function, the exception is thrown only at some documents.

I plug it into simple transformer + ner pipeline like this: https://github.com/tomateit/natasha-spacy/blob/transformer-pipeline/project/config_trf.cfg
(in my tests I disabled all but transformer and NER)
This error is emitted at the line https://github.com/explosion/spaCy/blob/master/spacy/ml/_precomputable_affine.py#L49

## Your Environment
- **spaCy version:** 3.2.0
- **Platform:** Linux-5.4.0-90-generic-x86_64-with-glibc2.29
- **Python version:** 3.8.10
- **spacy-transformers** = "^1.0.6"
- **torch** = "1.9.1"
* Operating System: Ubuntu 20.04
* Python Version Used: Python 3.8.10
* spaCy Version Used: 3.2.0

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.