explosion / explosion/spacy-stanza

Stanza's sentencizer only works when `processors = 'tokenize,pos,lemma,depparse'`

Open
#57 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
748
Forks
63
PR merge metrics
No merged PRs in 30d

Description

Hi all,

I started an NLP project where I needed high accuracy sentence segmentation, and therefore decided to use stanza.

I was thrilled to find this library, since Spacy is quite intuitive. However, I found that the sentence segmentation only gets carried into spacy under certain conditions.

**Baseline:**

The baseline text is to use the Stanza model alone to see if the sentence segmentation works.

This is the simplest model that I could use, I simply turned on the `tokenize` processor.

Screenshot 2021-02-03 at 18 57 31

**Test with Spacy-Stanza:**

I then tried the same thing, but this time added the spacy-stanza wrapper.

Screenshot 2021-02-03 at 18 58 00

As shown above, the sentences were not actually tokenized.

**Test with spacy-stanza with more processors on Stanza:**

Screenshot 2021-02-03 at 18 56 23

It seems that the `depparse` processor is necessary, but this is rather confusing since the vanilla stanza model does not require it to tokenize.

Any help would be appreciated :)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.