NER fails on warning "Token indices sequence length is longer than the specified maximum"
- Dominant language
- Python
- Stars
- 33.9k
- Forks
- 4.7k
- Avg merge
- 3m
- Merged PRs (30d)
- 1
Description
## How to reproduce the behaviour
code:
> text1="My name is robert" + " \r\n " * 1000 + "My name is robert"
> text2="My name is robert " + "contact" * 800 + " My name is robert"
> import spacy
> nlp = spacy.load("en_core_web_trf")
> doc = nlp(text1)
console output:
> Token indices sequence length is longer than the specified maximum sequence length for this model (4011 > 512). Running this sequence through the model will result in indexing errors
NER performance:
The NER performance of the last sentence degenerates:
> My ORDINAL
> name ORDINAL
> is ORDINAL
> robert ORDINAL
analysis:
When there are words or tokens that repeat massively, the warning is emitted. Unlike mentioned in other sources (https://github.com/explosion/spaCy/issues/6939) I think the impact is big. Before the occurence of the "big word" NER works normal but it fails on sentences directly following it. After some sentences the NER performance usually normalizes and works again. I am aware that spacy-transformers chunks big texts and thanks to that we can process long texts. But something is buggy and I think it has something to do with the chunking in the spacy-transformers library combined with the byte pair encoding used by the tokenizer of the trf model.
> for i in doc._.trf_data.to_dict()["wordpieces"]["strings"]:
> print(len(i))
> print(i)
As long as the bug exists, the warning should be emitted every time and not just once. Unfortunately I could not change that behavior.
## Your Environment
- **spaCy version:** 3.5.2
- **Platform:** Windows-10-10.0.19044-SP0
- **Python version:** 3.10.10
- **Pipelines:** en_core_web_trf (3.5.0)
Contributor guide
Assessment
This issue has not been assessed yet.