Dependency sentence segmenter handles newlines inconsistently between languages
- Dominant language
- Python
- Stars
- 33.9k
- Forks
- 4.7k
- Avg merge
- 3m
- Merged PRs (30d)
- 1
Description
## How to reproduce the behaviour
[Colab notebook demonstrating problem](https://colab.research.google.com/drive/14FFYKqjRVRbN7aAVmHUYEao9CwahY0We?usp=sharing)
When parsing a sentence that contains newlines, the Italian parser sometimes assigns the newline to a sentence by itself, for example:
>Ma regolamenta solo un settore, a differenza dell’azione a largo raggio dell’Inflation Act. \nI tentativi di legiferare per stimolare l’industria non hanno avuto molto successo.
Produces 3 sentences:
```
'Ma regolamenta solo un settore, a differenza dell’azione a largo raggio dell’Inflation Act (dalla sanità all’industria pesante).'
'\n'
'I tentativi di legiferare per stimolare l’industria non hanno avuto molto successo.'
```
There are various experiments with different combinations of punctuation in the notebook.
Looking at the tokens and their `is_sent_start` property, it seems under some circumstances the `\n` and `I` tokens are both assigned as the start of a new sentence.
I have not been able to cause this problem with `en_core_web_sm`, which always correctly identifies 2 sentences.
Although I understand that sentence segmentation based on the dependency parser is probabilistic and not always correct, it seems there's some inconsistency between languages here, and I don't think it would ever be correct for a whitespace token to be assigned as the start of a sentence.
## Your Environment
- **spaCy version:** 3.6.1
- **Platform:** Linux-5.15.120+-x86_64-with-glibc2.35
- **Python version:** 3.10.12
- **Pipelines:** it_core_news_sm (3.6.0), en_core_web_sm (3.6.0)
Contributor guide
Assessment
This issue has not been assessed yet.