Doc span group spans aren't adjusted for retokenization
- Dominant language
- Python
- Stars
- 33.9k
- Forks
- 4.7k
- Avg merge
- 3m
- Merged PRs (30d)
- 1
Description
When a [Doc](https://spacy.io/api/doc) object is retokenized, the entity spans in [Doc.ents](https://spacy.io/api/doc#ents) reflect the new token alignment, but the spans in [Doc.spans](https://spacy.io/api/doc#spans) retain the original doc's token indexes, leading to unexpected spans.
## How to reproduce the behaviour
Here is a modification of the example for [Retokenizer.split](https://spacy.io/api/doc#spans) that stores `NewYork` as an entity and in a span group before retokenization.
**Split Example Code:**
```python
import spacy
from spacy.tokens import Span
nlp = spacy.blank("en")
doc = nlp("I live in NewYork")
ny = Span(doc, 3, 4, label="CITY")
doc.ents = [ny]
doc.spans["spans"] = [ny]
with doc.retokenize() as retokenizer:
heads = [(doc[3], 1), doc[2]]
attrs = {"POS": ["PROPN", "PROPN"],
"DEP": ["pobj", "compound"]}
retokenizer.split(doc[3], ["New", "York"], heads=heads, attrs=attrs)
```
**Output:**
```python
>>> print(doc.ents)
(NewYork,)
>>> print(doc.spans["spans"])
[New]
```
Notice how the entity span is still `NewYork`, but the span group span is now just `New`.
In the case of a merge, spans can be lost altogether by exceeding the range of the retokenized doc. Here is a modification of the example for [Retokenizer.merge](https://spacy.io/api/doc#retokenizer.merge).
**Merge Example Code:**
```python
doc = nlp("I like David Bowie")
db = Span(doc, 2, 4, label="PERSON")
doc.ents = [db]
doc.spans["spans"] = [db]
with doc.retokenize() as retokenizer:
attrs = {"LEMMA": "David Bowie"}
retokenizer.merge(doc[2:4], attrs=attrs)
print(doc.spans["spans"])
print(doc.ents)
```
**Output:**
```python
>>> print(doc.ents)
(David Bowie,)
>>> print(doc.spans["spans"])
[]
```
The range of the original `David Bowie` span is no longer valid and disappears from the span group.
## Your Environment
* Operating System: macOS Ventura 13.1 (22C65)
* Python Version Used: 3.11.0
* spaCy Version Used: 3.4.4
* Environment Information:
Contributor guide
Assessment
This issue has not been assessed yet.