Batch processing does not speed up `en_core_web_trf`
- Dominant language
- Python
- Stars
- 33.9k
- Forks
- 4.7k
- Avg merge
- 3m
- Merged PRs (30d)
- 1
Description
## How to reproduce the behaviour
```python
spacy.prefer_gpu()
nlp = spacy.load(
"en_core_web_trf",
disable=['tagger', 'ner', 'lemmatizer', 'textcat']
)
node = """Some really long string, 3000 characters"""
# simulating 96 pretty long docs
nodes = [node*25]*96
```
Then, run each of the below lines separately and time it:
```
# 1 minute 7.5 s
[list(doc.sents) for doc in nlp.pipe(nodes, batch_size=96)]
# 1 minute 7.3 s
[list(doc.sents) for doc in nlp.pipe(nodes, batch_size=32)]
# 1 m 8.2 s
[list(doc.sents) for doc in nlp.pipe(nodes, batch_size=1)]
```
Running the same thing with `en_core_web_lg` results in substantial gains due to batching. Largest batch size is roughly 1/4 of the runtime of batch_size=1.
## Your Environment
Using a single RTX A6000
python -m spacy info --markdown:
## Info about spaCy
- **spaCy version:** 3.7.4
- **Platform:** Linux-5.15.0-94-generic-x86_64-with-glibc2.35
- **Python version:** 3.10.12
- **Pipelines:** en_core_web_lg (3.7.1), en_core_web_trf (3.7.3), en_core_web_sm (3.7.1), de_core_news_sm (3.7.0)
## Expected Behavior
My understanding from the documentation and this [issue](https://github.com/explosion/spaCy/issues/4935) is that we should expect significant gains from batching, as observed with `en_core_web_lg`. However, using `en_core_web_trf` does not yield significant gains from batching.
I'm wondering if this is a bug, or if we should not expect improved performance due to batching for a Transformer-Parser pipeline. Thanks for this awesome package, and in advance for your help!
Contributor guide
Research direction
Start by running the provided `nlp.pipe` benchmark with `en_core_web_trf` at batch sizes 1, 32, and 96, then compare it with `en_core_web_lg` using the listed spaCy and pipeline versions. Trace the `nlp.pipe` transformer path to determine whether batching is expected to improve this workload. Done means explaining the behavior and correcting it if it is unintended.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100