explosion / explosion/spaCy

Lowercase lemmatization in pipe, when tok2vec disabled

Open
#13,511 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
33.9k
Forks
4.7k
Avg merge
3m
Merged PRs (30d)
1

Description

## Introduction
Hi! I am using spaCy lemmatizer for some tasks. I saw that when using a pipe to process the data faster, I'm getting different results with tok2vec disabled or enabled. Maintaining case-sensitivity is critical for me. Is the below behavior expected?

## How to reproduce the behaviour
### Case1
```python
import spacy
nlp = spacy.load("en_core_web_sm")

for doc in nlp.pipe(["Hello! My name is Marcin.", "I have a SFTP server running in my HomeLab"], batch_size=100, n_process=1, disable=["ner", "tok2vec"]):
for token in doc:
print(str(token), token.lemma_)
print("")
```
Output:
```
Hello hello
! !
My my
name name
is is
Marcin marcin
. .

I i
have have
a a
SFTP sftp
server server
running running
in in
my my
HomeLab homelab
```

### Case2
```python
import spacy
nlp = spacy.load("en_core_web_sm")

for doc in nlp.pipe(["Hello! My name is Marcin.", "I have a SFTP server running in my HomeLab"], batch_size=100, n_process=1, disable=["ner"]):
for token in doc:
print(str(token), token.lemma_)
print("")
```
Output:
```
Hello hello
! !
My my
name name
is be
Marcin Marcin
. .

I I
have have
a a
SFTP sftp
server server
running run
in in
my my
HomeLab HomeLab
```

## Info about spaCy
- **spaCy version:** 3.7.2
- **Platform:** Windows-10-10.0.19045-SP0
- **Python version:** 3.10.13
- **Pipelines:** en_core_web_sm (3.7.1), en_core_web_trf (3.7.3), es_core_news_lg (3.7.0), es_core_news_sm (3.7.0), pl_core_news_lg (3.7.0), pl_core_news_sm (3.7.0)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.