explosion / explosion/spaCy

`Spacy` has inconsistency when dividing sentences

Open
#13,346 5 comments 0 reactions 0 assignees View on GitHub
feat / parser
Dominant language
Python
Stars
33.9k
Forks
4.7k
Avg merge
3m
Merged PRs (30d)
1

Description

Hello,

I am using `Spacy` to divide sentences after joining a set of words with whitespaces. But to my dismay, this process has unpredictable and unexplainable behaviour. I have a custom segmentation function where I am trying to set custom sentence boundaries (ie `is_sent_start`).

Custom Function:
```py
from spacy.language import Language

@Language.component("segm")
def set_custom_segmentation(doc):
i = 0
while i < len(doc[:-1]):
if doc[i].text.lower() in ["eq", "fig", "al", 'table', "fig."]:
doc[i+1].is_sent_start = False
i+=1
elif doc[i].text in ["(", "'s"]:
doc[i].is_sent_start = False
i+=1
elif doc[i].text in [".", ")."]:
doc[i+1].is_sent_start = True
else:
doc[i+1].is_sent_start = False
i+=1
return doc

nlp = spacy.load('en_core_web_sm')
nlp.add_pipe("segm", before="parser")
nlp.pipeline
```

This is my `nlp.pipeline`.
```py
[('tok2vec', ),
('tagger', ),
('segm', ),
('parser', ),
('attribute_ruler',
),
('lemmatizer', ),
('ner', )]
```
## How to reproduce the behaviour

```py
doc = nlp("Massive ETGs are summarized in a schematic way in Fig. 2 . ##(this is the sentence to consider)## We refer the reader to fig. 1 of Forbes et al. ( 2011 ) and fig. 10 of Faifer et al. ( 2011 ) for real-world examples of our schematic plot, which show not only the mean gradients but also the individual GC data points. Figure 2.")

for sent in doc.sents:
print(sent)
```

This is the current output:
Screenshot 2024-02-22 at 13 50 02

The form of the tokens here `Fig. 2 .` produces different outputs for the sentences. Please see the following examples.
1. Here if we change the `Massive ETGs are summarized in a schematic way in Fig. 21 .` _(changed 2 . to 21 . )_
Screenshot 2024-02-22 at 13 50 37

2. Here if we change the `Massive ETGs are summarized in a schematic way in Fig. 21.` _(removed space b/w 21 & period)_
Screenshot 2024-02-22 at 13 51 41

3. Here if we change the `Massive ETGs are summarized in a schematic way in Fig. 2.` _(removed space b/w 2 & period)_
Screenshot 2024-02-22 at 13 52 08

4. Here if we change the `Massive ETGs are summarized in a schematic way in Fig. 1 .` _(changed 2 to 1)_
Screenshot 2024-02-22 at 13 53 48

5. Here if we change the `Massive ETGs are summarized in a schematic way in Fig. 3 .` _(changed 2 to 3)_
Screenshot 2024-02-22 at 13 54 26

6. Here if we change the `Massive ETGs are summarized in a schematic way in Fig. 4 .` _(changed 2 to 4)_
Screenshot 2024-02-22 at 13 54 41

7. Here if we change the `Massive ETGs are summarized in a schematic way in Fig. 4.` _(changed 2 to 4 and removed whitespace)_
Screenshot 2024-02-22 at 13 55 04

8. Here if we change the `Massive ETGs are summarized in a schematic way in Fig. 200.` _(changed 2 to 200 and removed space)_
Screenshot 2024-02-22 at 13 55 43

9. Here if we change the `Massive ETGs are summarized in a schematic way in Fig. 200 .` _(changed 2 to 200)_
Screenshot 2024-02-22 at 13 56 19

There is inconsistent behaviour in the way the sentence boundaries are categorised here. I have other examples as well so if needed I can share them here.

Any help in understanding this would be appreciated.

## Your Environment

- **spaCy version:** 3.6.0
- **Platform:** macOS-14.3.1-arm64-arm-64bit
- **Python version:** 3.10.12
- **Pipelines:** en_core_web_lg (3.6.0), en_core_web_sm (3.6.0)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.