explosion / explosion/spaCy

NER fails on warning "Token indices sequence length is longer than the specified maximum"

Open
#12,622 3 comments 0 reactions 0 assignees View on GitHub
perf / accuracy
Dominant language
Python
Stars
33.9k
Forks
4.7k
Avg merge
3m
Merged PRs (30d)
1

Description

## How to reproduce the behaviour
code:
> text1="My name is robert" + " \r\n " * 1000 + "My name is robert"
> text2="My name is robert " + "contact" * 800 + " My name is robert"
> import spacy
> nlp = spacy.load("en_core_web_trf")
> doc = nlp(text1)

console output:
> Token indices sequence length is longer than the specified maximum sequence length for this model (4011 > 512). Running this sequence through the model will result in indexing errors

NER performance:
The NER performance of the last sentence degenerates:

> My ORDINAL
> name ORDINAL
> is ORDINAL
> robert ORDINAL

analysis:
When there are words or tokens that repeat massively, the warning is emitted. Unlike mentioned in other sources (https://github.com/explosion/spaCy/issues/6939) I think the impact is big. Before the occurence of the "big word" NER works normal but it fails on sentences directly following it. After some sentences the NER performance usually normalizes and works again. I am aware that spacy-transformers chunks big texts and thanks to that we can process long texts. But something is buggy and I think it has something to do with the chunking in the spacy-transformers library combined with the byte pair encoding used by the tokenizer of the trf model.

> for i in doc._.trf_data.to_dict()["wordpieces"]["strings"]:
> print(len(i))
> print(i)

As long as the bug exists, the warning should be emitted every time and not just once. Unfortunately I could not change that behavior.

## Your Environment
- **spaCy version:** 3.5.2
- **Platform:** Windows-10-10.0.19044-SP0
- **Python version:** 3.10.10
- **Pipelines:** en_core_web_trf (3.5.0)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.