explosion / explosion/spaCy

Suffix doesn't match for sentence ending in uppercase.

Open
#6,695 3 comments 0 reactions 0 assignees View on GitHub
feat / tokenizer lang / en
Dominant language
Python
Stars
33.9k
Forks
4.7k
Avg merge
3m
Merged PRs (30d)
1

Description

## How to reproduce the behaviour
```
import spacy
nlp = spacy.load("en_core_web_sm")
list(nlp.tokenizer("about the P&L."))
```

I get

```
[about, the, P&L.]
```

The . should be separated from P&L here.

This behaviour comes from, https://github.com/explosion/spaCy/blob/bf778f59c7ea48787ef4aac79ca2f1e33fe33e08/spacy/lang/punctuation.py#L33

the requirement for double uppercase is likely for acronyms but perhaps an ampersand is acceptable.

eg ` r"(?<=&[{au}])\.".format(au=ALPHA_UPPER)`

## Your Environment

* **spaCy version:** 2.3.2
* **Platform:** Darwin-19.6.0-x86_64-i386-64bit
* **Python version:** 3.6.12

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.