explosion / explosion/spaCy

IndexError E040 when using senter

Open
#10,801 3 comments 0 reactions 0 assignees View on GitHub
feat / doc
Dominant language
Python
Stars
33.9k
Forks
4.7k
Avg merge
3m
Merged PRs (30d)
1

Description

## How to reproduce the behaviour
As suggested on this page (https://spacy.io/models) I can replace the "parser" by the "senter" as a more efficient way to detect sentence boundaries. I did so, but ran into an E040 error when printing the individual noun_chunks of the document (which is not happening when using the original parser).

Code to reproduce:
```
nlp = spacy.load("en_core_web_sm")
nlp.disable_pipe('parser')
nlp.enable_pipe('senter')
doc = nlp(text)

for chunk in doc.noun_chunks:
print(chunk)
```

Error message:
```
File "spacy\tokens\token.pyx", line 609, in spacy.tokens.token.Token.left_edge.__get__
File "spacy\tokens\doc.pyx", line 474, in spacy.tokens.doc.Doc.__getitem__
File "spacy\tokens\token.pxd", line 23, in spacy.tokens.token.Token.cinit
IndexError: [E040] Attempt to access token at 10794, max length 10792.
```
## Your Environment
- **spaCy version:** 3.2.3
- **Platform:** Windows-10-10.0.19042-SP0
- **Python version:** 3.9.7
- **Pipelines:** en_core_web_lg (3.2.0), en_core_web_md (3.2.0), en_core_web_sm (3.2.0), en_core_web_trf (3.2.0)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.