explosion / explosion/spaCy

Doc span group spans aren't adjusted for retokenization

Open
#12,024 4 comments 0 reactions 0 assignees View on GitHub
bug feat / doc feat / tokenizer
Dominant language
Python
Stars
33.9k
Forks
4.7k
Avg merge
3m
Merged PRs (30d)
1

Description

When a [Doc](https://spacy.io/api/doc) object is retokenized, the entity spans in [Doc.ents](https://spacy.io/api/doc#ents) reflect the new token alignment, but the spans in [Doc.spans](https://spacy.io/api/doc#spans) retain the original doc's token indexes, leading to unexpected spans.

## How to reproduce the behaviour

Here is a modification of the example for [Retokenizer.split](https://spacy.io/api/doc#spans) that stores `NewYork` as an entity and in a span group before retokenization.

**Split Example Code:**

```python
import spacy
from spacy.tokens import Span

nlp = spacy.blank("en")

doc = nlp("I live in NewYork")
ny = Span(doc, 3, 4, label="CITY")
doc.ents = [ny]
doc.spans["spans"] = [ny]

with doc.retokenize() as retokenizer:
heads = [(doc[3], 1), doc[2]]
attrs = {"POS": ["PROPN", "PROPN"],
"DEP": ["pobj", "compound"]}
retokenizer.split(doc[3], ["New", "York"], heads=heads, attrs=attrs)
```

**Output:**

```python
>>> print(doc.ents)
(NewYork,)
>>> print(doc.spans["spans"])
[New]
```

Notice how the entity span is still `NewYork`, but the span group span is now just `New`.

In the case of a merge, spans can be lost altogether by exceeding the range of the retokenized doc. Here is a modification of the example for [Retokenizer.merge](https://spacy.io/api/doc#retokenizer.merge).

**Merge Example Code:**

```python
doc = nlp("I like David Bowie")
db = Span(doc, 2, 4, label="PERSON")
doc.ents = [db]
doc.spans["spans"] = [db]

with doc.retokenize() as retokenizer:
attrs = {"LEMMA": "David Bowie"}
retokenizer.merge(doc[2:4], attrs=attrs)

print(doc.spans["spans"])
print(doc.ents)
```

**Output:**

```python
>>> print(doc.ents)
(David Bowie,)
>>> print(doc.spans["spans"])
[]
```

The range of the original `David Bowie` span is no longer valid and disappears from the span group.

## Your Environment

* Operating System: macOS Ventura 13.1 (22C65)
* Python Version Used: 3.11.0
* spaCy Version Used: 3.4.4
* Environment Information:

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.