explosion / explosion/spaCy

Batch processing does not speed up `en_core_web_trf`

Open
#13,500 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
33.9k
Forks
4.7k
Avg merge
3m
Merged PRs (30d)
1

Description

## How to reproduce the behaviour
```python
spacy.prefer_gpu()
nlp = spacy.load(
"en_core_web_trf",
disable=['tagger', 'ner', 'lemmatizer', 'textcat']
)

node = """Some really long string, 3000 characters"""

# simulating 96 pretty long docs
nodes = [node*25]*96
```
Then, run each of the below lines separately and time it:
```
# 1 minute 7.5 s
[list(doc.sents) for doc in nlp.pipe(nodes, batch_size=96)]

# 1 minute 7.3 s
[list(doc.sents) for doc in nlp.pipe(nodes, batch_size=32)]

# 1 m 8.2 s
[list(doc.sents) for doc in nlp.pipe(nodes, batch_size=1)]
```
Running the same thing with `en_core_web_lg` results in substantial gains due to batching. Largest batch size is roughly 1/4 of the runtime of batch_size=1.

## Your Environment

Using a single RTX A6000

python -m spacy info --markdown:
## Info about spaCy

- **spaCy version:** 3.7.4
- **Platform:** Linux-5.15.0-94-generic-x86_64-with-glibc2.35
- **Python version:** 3.10.12
- **Pipelines:** en_core_web_lg (3.7.1), en_core_web_trf (3.7.3), en_core_web_sm (3.7.1), de_core_news_sm (3.7.0)

## Expected Behavior
My understanding from the documentation and this [issue](https://github.com/explosion/spaCy/issues/4935) is that we should expect significant gains from batching, as observed with `en_core_web_lg`. However, using `en_core_web_trf` does not yield significant gains from batching.

I'm wondering if this is a bug, or if we should not expect improved performance due to batching for a Transformer-Parser pipeline. Thanks for this awesome package, and in advance for your help!

Contributor guide

Open the contributing guide

Research direction

Start by running the provided `nlp.pipe` benchmark with `en_core_web_trf` at batch sizes 1, 32, and 96, then compare it with `en_core_web_lg` using the listed spaCy and pipeline versions. Trace the `nlp.pipe` transformer path to determine whether batching is expected to improve this workload. Done means explaining the behavior and correcting it if it is unintended.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.