explosion / explosion/spaCy

Dependency sentence segmenter handles newlines inconsistently between languages

Open
#13,059 3 comments 0 reactions 0 assignees View on GitHub
feat / senter lang / it
Dominant language
Python
Stars
33.9k
Forks
4.7k
Avg merge
3m
Merged PRs (30d)
1

Description

## How to reproduce the behaviour
[Colab notebook demonstrating problem](https://colab.research.google.com/drive/14FFYKqjRVRbN7aAVmHUYEao9CwahY0We?usp=sharing)

When parsing a sentence that contains newlines, the Italian parser sometimes assigns the newline to a sentence by itself, for example:

>Ma regolamenta solo un settore, a differenza dell’azione a largo raggio dell’Inflation Act. \nI tentativi di legiferare per stimolare l’industria non hanno avuto molto successo.

Produces 3 sentences:
```
'Ma regolamenta solo un settore, a differenza dell’azione a largo raggio dell’Inflation Act (dalla sanità all’industria pesante).'
'\n'
'I tentativi di legiferare per stimolare l’industria non hanno avuto molto successo.'
```
There are various experiments with different combinations of punctuation in the notebook.

Looking at the tokens and their `is_sent_start` property, it seems under some circumstances the `\n` and `I` tokens are both assigned as the start of a new sentence.

I have not been able to cause this problem with `en_core_web_sm`, which always correctly identifies 2 sentences.

Although I understand that sentence segmentation based on the dependency parser is probabilistic and not always correct, it seems there's some inconsistency between languages here, and I don't think it would ever be correct for a whitespace token to be assigned as the start of a sentence.

## Your Environment
- **spaCy version:** 3.6.1
- **Platform:** Linux-5.15.120+-x86_64-with-glibc2.35
- **Python version:** 3.10.12
- **Pipelines:** it_core_news_sm (3.6.0), en_core_web_sm (3.6.0)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.