explosion / explosion/spaCy

Tokenizer special cases do not work around infix punctuation

Open
#5,598 3 comments 0 reactions 0 assignees View on GitHub
enhancement feat / tokenizer lang / en
Dominant language
Python
Stars
33.9k
Forks
4.7k
Avg merge
3m
Merged PRs (30d)
1

Description

## How to reproduce the behaviour
I would expect the two sentences below to be tokenized the same way. However, in the second, the special cases for "won't" and "can't" do not work.

>>> import en_core_web_sm
>>> nlp = en_core_web_sm.load()
>>> [token.text for token in nlp("I can't / won't tolerate that.")]
['I', 'ca', "n't", '/', 'wo', "n't", 'tolerate', 'that', '.']
>>> [token.text for token in nlp("I can't/won't tolerate that.")]
['I', "can't", '/', "won't", 'tolerate', 'that', '.']

## Your Environment

* **spaCy version:** 2.3.0
* **Platform:** Darwin-18.7.0-x86_64-i386-64bit
* **Python version:** 3.7.4

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.