explosion / explosion/spaCy

`nlp.pipe(..., n_process>1)` won't return if wrapped by `tqdm()` and `zip()`

Open
#8,798 4 comments 0 reactions 0 assignees View on GitHub
bug scaling
Dominant language
Python
Stars
33.9k
Forks
4.7k
Avg merge
3m
Merged PRs (30d)
1

Description

This is a pretty specific set of circumstances that I discovered today, but on the off chance that it is useful to someone, here it is.

If you include the output of `nlp.pipe(...,n_process>1)` in a `zip()` within `tqdm()` it will hang interminably. See below

## How to reproduce the behaviour
```
#!/usr/bin/env python

import pandas as pd
import spacy
from tqdm import tqdm

data = [
{"text": "I just wanna tell you how I'm feeling", "id": 0},
{"text": "Gotta make you understand", "id": 1},
{"text": "Never gonna give you up", "id": 2},
{"text": "Never gonna let you down", "id": 3},
{"text": "Never gonna run around and desert you", "id": 4},
{"text": "Never gonna make you cry", "id": 5},
{"text": "Never gonna say goodbye", "id": 6},
{"text": "Never gonna tell a lie and hurt you", "id": 7},
]

df = pd.DataFrame(data)

nlp = spacy.load("en_core_web_md")

# Works with a single process

for id, doc in tqdm(
zip(
df["id"],
nlp.pipe(
df["text"],
n_process=1,
),
)
):

print(id, doc.text)

# Works with no zip and multiple processes

for doc in tqdm(
nlp.pipe(
df["text"],
n_process=2,
),
):

print(doc.text)

# Hangs with multiple processes and zip

for id, doc in tqdm(
zip(
df["id"],
nlp.pipe(
df["text"],
n_process=2,
),
)
):

print(id, doc.text)
```
Output:

```
$python script.py
0it [00:00, ?it/s]0 I just wanna tell you how I'm feeling
1 Gotta make you understand
2 Never gonna give you up
3 Never gonna let you down
4 Never gonna run around and desert you
5 Never gonna make you cry
6 Never gonna say goodbye
7 Never gonna tell a lie and hurt you
8it [00:00, 977.21it/s]
0it [00:00, ?it/s]I just wanna tell you how I'm feeling
Gotta make you understand
Never gonna give you up
Never gonna let you down
Never gonna run around and desert you
Never gonna make you cry
Never gonna say goodbye
Never gonna tell a lie and hurt you
8it [00:00, 345.82it/s]
0it [00:00, ?it/s]0 I just wanna tell you how I'm feeling
1 Gotta make you understand
2 Never gonna give you up
3 Never gonna let you down
4 Never gonna run around and desert you
5 Never gonna make you cry
6 Never gonna say goodbye
7 Never gonna tell a lie and hurt you
8it [00:00, 342.59it/s]

```

## Your Environment
## Info about spaCy

- **spaCy version:** 3.0.1
- **Platform:** Linux-5.8.0-50-generic-x86_64-with-glibc2.29
- **Python version:** 3.8.5
- **Pipelines:** en_ner_bc5cdr_md (0.4.0), en_core_web_md (3.0.0), en_ner_craft_md (0.4.0), en_ner_bionlp13cg_md (0.4.0), en_ner_jnlpba_md (0.4.0)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.