Tokenizer special cases do not work around infix punctuation
Open
enhancement
feat / tokenizer
lang / en
- Dominant language
- Python
- Stars
- 33.9k
- Forks
- 4.7k
- Avg merge
- 3m
- Merged PRs (30d)
- 1
Description
## How to reproduce the behaviour
I would expect the two sentences below to be tokenized the same way. However, in the second, the special cases for "won't" and "can't" do not work.
>>> import en_core_web_sm
>>> nlp = en_core_web_sm.load()
>>> [token.text for token in nlp("I can't / won't tolerate that.")]
['I', 'ca', "n't", '/', 'wo', "n't", 'tolerate', 'that', '.']
>>> [token.text for token in nlp("I can't/won't tolerate that.")]
['I', "can't", '/', "won't", 'tolerate', 'that', '.']
## Your Environment
* **spaCy version:** 2.3.0
* **Platform:** Darwin-18.7.0-x86_64-i386-64bit
* **Python version:** 3.7.4
Contributor guide
Assessment
This issue has not been assessed yet.