Handle sentence boundaries from multiple components
- Dominant language
- Python
- Stars
- 33.9k
- Forks
- 4.7k
- Avg merge
- 3m
- Merged PRs (30d)
- 1
Description
## Feature description
Decide how to handle `is_sentenced` and sentence boundaries that may come from multiple components (Sentencizer, SentenceRecognizer, Parser).
Some ideas:
* have an `is_sentenced` property more like `is_parsed` that can be set by components
* have a way to set finalized sentence boundaries (all 0 to -1):
* have an extra option for each component
* have an extra pipeline component (e.g., `finalize_sentences`?) that can be inserted at the right point in the pipeline
* also have a component that resets all sentence boundaries?
* modify Sentencizer to only set sentence starts, not all tokens?
Check that no spacy components clobber sentence boundaries and that `is_sentenced` works consistently when sentence boundaries come from multiple sources. If a component after the parser changes sentence boundaries, make sure the required tree recalculations are done (a related issue: #4497).
Potentially add warnings when non-zero `sent_start` is changed by any component?
I think the default behavior could be that any pipeline component can add sentence boundaries but that components won't remove any sentence boundaries. The idea would be that the Sentencizer or SentenceRecognizer add punctuation-based boundaries (typically high precision, although the Sentencizer less so) and the Parser can add phrase-based boundaries (improving recall). I don't know if this works as cleanly as envisioned in practice, especially with the Sentencizer. Most likely people using the Sentencizer aren't using other components so it's less of an issue, but I could imagine SentenceRecognizer + Parser as a common combination.
Contributor guide
Assessment
This issue has not been assessed yet.