explosion / explosion/spaCy

Spaces impacting tag/pos

Open
#13,680 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
33.9k
Forks
4.7k
Avg merge
3m
Merged PRs (30d)
1

Description

## How to reproduce the behaviour

Notice the double space in front of `sourire` in the first case vs. the single space in the second case

`Les publics avec un sourire chaleureux et`

image

https://demos.explosion.ai/displacy?text=Les%20publics%20avec%20un%20%20sourire%20chaleureux%20%20et&model=fr_core_news_sm

vs.

`Les publics avec un sourire chaleureux et`

image

https://demos.explosion.ai/displacy?text=Les%20publics%20avec%20un%20sourire%20chaleureux%20%20et&model=fr_core_news_sm

## Your Environment

* Operating System:
* Python Version Used: 3.12
* spaCy Version Used: v3.5 (displacy) but also in v3.7
* Environment Information:

Semi-related: Any guidance on how to modify the tokenizer so that a double spaces would be placed into `whitespace_` (ie. ` `) and not lead to a `SPACE` token? I did take note of https://github.com/explosion/spaCy/issues/1707 though putting the additional spaces into `whitespace_` seems more logical to me.

## Research

a) Maybe related https://github.com/explosion/spaCy/issues/621
b) Semi-related https://stephantul.github.io/spacy/2019/05/01/tokenizationspacy/
c) Semi-related https://github.com/explosion/spaCy/discussions/9978

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.