explosion / explosion/spaCy

Handle sentence boundaries from multiple components

Open
#4,775 3 comments 0 reactions 0 assignees View on GitHub
enhancement feat / doc feat / parser feat / sentencizer
Dominant language
Python
Stars
33.9k
Forks
4.7k
Avg merge
3m
Merged PRs (30d)
1

Description

## Feature description

Decide how to handle `is_sentenced` and sentence boundaries that may come from multiple components (Sentencizer, SentenceRecognizer, Parser).

Some ideas:

* have an `is_sentenced` property more like `is_parsed` that can be set by components
* have a way to set finalized sentence boundaries (all 0 to -1):
* have an extra option for each component
* have an extra pipeline component (e.g., `finalize_sentences`?) that can be inserted at the right point in the pipeline
* also have a component that resets all sentence boundaries?
* modify Sentencizer to only set sentence starts, not all tokens?

Check that no spacy components clobber sentence boundaries and that `is_sentenced` works consistently when sentence boundaries come from multiple sources. If a component after the parser changes sentence boundaries, make sure the required tree recalculations are done (a related issue: #4497).

Potentially add warnings when non-zero `sent_start` is changed by any component?

I think the default behavior could be that any pipeline component can add sentence boundaries but that components won't remove any sentence boundaries. The idea would be that the Sentencizer or SentenceRecognizer add punctuation-based boundaries (typically high precision, although the Sentencizer less so) and the Parser can add phrase-based boundaries (improving recall). I don't know if this works as cleanly as envisioned in practice, especially with the Sentencizer. Most likely people using the Sentencizer aren't using other components so it's less of an issue, but I could imagine SentenceRecognizer + Parser as a common combination.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.