`Spacy` has inconsistency when dividing sentences
- Dominant language
- Python
- Stars
- 33.9k
- Forks
- 4.7k
- Avg merge
- 3m
- Merged PRs (30d)
- 1
Description
Hello,
I am using `Spacy` to divide sentences after joining a set of words with whitespaces. But to my dismay, this process has unpredictable and unexplainable behaviour. I have a custom segmentation function where I am trying to set custom sentence boundaries (ie `is_sent_start`).
Custom Function:
```py
from spacy.language import Language
@Language.component("segm")
def set_custom_segmentation(doc):
i = 0
while i < len(doc[:-1]):
if doc[i].text.lower() in ["eq", "fig", "al", 'table', "fig."]:
doc[i+1].is_sent_start = False
i+=1
elif doc[i].text in ["(", "'s"]:
doc[i].is_sent_start = False
i+=1
elif doc[i].text in [".", ")."]:
doc[i+1].is_sent_start = True
else:
doc[i+1].is_sent_start = False
i+=1
return doc
nlp = spacy.load('en_core_web_sm')
nlp.add_pipe("segm", before="parser")
nlp.pipeline
```
This is my `nlp.pipeline`.
```py
[('tok2vec', ),
('tagger', ),
('segm', ),
('parser', ),
('attribute_ruler',
),
('lemmatizer', ),
('ner', )]
```
## How to reproduce the behaviour
```py
doc = nlp("Massive ETGs are summarized in a schematic way in Fig. 2 . ##(this is the sentence to consider)## We refer the reader to fig. 1 of Forbes et al. ( 2011 ) and fig. 10 of Faifer et al. ( 2011 ) for real-world examples of our schematic plot, which show not only the mean gradients but also the individual GC data points. Figure 2.")
for sent in doc.sents:
print(sent)
```
This is the current output:
The form of the tokens here `Fig. 2 .` produces different outputs for the sentences. Please see the following examples.
1. Here if we change the `Massive ETGs are summarized in a schematic way in Fig. 21 .` _(changed 2 . to 21 . )_
2. Here if we change the `Massive ETGs are summarized in a schematic way in Fig. 21.` _(removed space b/w 21 & period)_
3. Here if we change the `Massive ETGs are summarized in a schematic way in Fig. 2.` _(removed space b/w 2 & period)_
4. Here if we change the `Massive ETGs are summarized in a schematic way in Fig. 1 .` _(changed 2 to 1)_
5. Here if we change the `Massive ETGs are summarized in a schematic way in Fig. 3 .` _(changed 2 to 3)_
6. Here if we change the `Massive ETGs are summarized in a schematic way in Fig. 4 .` _(changed 2 to 4)_
7. Here if we change the `Massive ETGs are summarized in a schematic way in Fig. 4.` _(changed 2 to 4 and removed whitespace)_
8. Here if we change the `Massive ETGs are summarized in a schematic way in Fig. 200.` _(changed 2 to 200 and removed space)_
9. Here if we change the `Massive ETGs are summarized in a schematic way in Fig. 200 .` _(changed 2 to 200)_
There is inconsistent behaviour in the way the sentence boundaries are categorised here. I have other examples as well so if needed I can share them here.
Any help in understanding this would be appreciated.
## Your Environment
- **spaCy version:** 3.6.0
- **Platform:** macOS-14.3.1-arm64-arm-64bit
- **Python version:** 3.10.12
- **Pipelines:** en_core_web_lg (3.6.0), en_core_web_sm (3.6.0)
Contributor guide
Assessment
This issue has not been assessed yet.