`nlp.pipe(..., n_process>1)` won't return if wrapped by `tqdm()` and `zip()`
- Dominant language
- Python
- Stars
- 33.9k
- Forks
- 4.7k
- Avg merge
- 3m
- Merged PRs (30d)
- 1
Description
This is a pretty specific set of circumstances that I discovered today, but on the off chance that it is useful to someone, here it is.
If you include the output of `nlp.pipe(...,n_process>1)` in a `zip()` within `tqdm()` it will hang interminably. See below
## How to reproduce the behaviour
```
#!/usr/bin/env python
import pandas as pd
import spacy
from tqdm import tqdm
data = [
{"text": "I just wanna tell you how I'm feeling", "id": 0},
{"text": "Gotta make you understand", "id": 1},
{"text": "Never gonna give you up", "id": 2},
{"text": "Never gonna let you down", "id": 3},
{"text": "Never gonna run around and desert you", "id": 4},
{"text": "Never gonna make you cry", "id": 5},
{"text": "Never gonna say goodbye", "id": 6},
{"text": "Never gonna tell a lie and hurt you", "id": 7},
]
df = pd.DataFrame(data)
nlp = spacy.load("en_core_web_md")
# Works with a single process
for id, doc in tqdm(
zip(
df["id"],
nlp.pipe(
df["text"],
n_process=1,
),
)
):
print(id, doc.text)
# Works with no zip and multiple processes
for doc in tqdm(
nlp.pipe(
df["text"],
n_process=2,
),
):
print(doc.text)
# Hangs with multiple processes and zip
for id, doc in tqdm(
zip(
df["id"],
nlp.pipe(
df["text"],
n_process=2,
),
)
):
print(id, doc.text)
```
Output:
```
$python script.py
0it [00:00, ?it/s]0 I just wanna tell you how I'm feeling
1 Gotta make you understand
2 Never gonna give you up
3 Never gonna let you down
4 Never gonna run around and desert you
5 Never gonna make you cry
6 Never gonna say goodbye
7 Never gonna tell a lie and hurt you
8it [00:00, 977.21it/s]
0it [00:00, ?it/s]I just wanna tell you how I'm feeling
Gotta make you understand
Never gonna give you up
Never gonna let you down
Never gonna run around and desert you
Never gonna make you cry
Never gonna say goodbye
Never gonna tell a lie and hurt you
8it [00:00, 345.82it/s]
0it [00:00, ?it/s]0 I just wanna tell you how I'm feeling
1 Gotta make you understand
2 Never gonna give you up
3 Never gonna let you down
4 Never gonna run around and desert you
5 Never gonna make you cry
6 Never gonna say goodbye
7 Never gonna tell a lie and hurt you
8it [00:00, 342.59it/s]
```
## Your Environment
## Info about spaCy
- **spaCy version:** 3.0.1
- **Platform:** Linux-5.8.0-50-generic-x86_64-with-glibc2.29
- **Python version:** 3.8.5
- **Pipelines:** en_ner_bc5cdr_md (0.4.0), en_core_web_md (3.0.0), en_ner_craft_md (0.4.0), en_ner_bionlp13cg_md (0.4.0), en_ner_jnlpba_md (0.4.0)
Contributor guide
Assessment
This issue has not been assessed yet.