ish-app / ish-app/ish

Python hangs fetching data with urllib in parallel

Open
#1,735 3 comments 0 reactions 0 assignees View on GitHub
unconfirmed
Dominant language
C
Stars
20.5k
Forks
1.5k
PR merge metrics
No merged PRs in 30d

Description

```
import time
from urllib import request
from queue import Queue
from threading import Thread

slots = 4

def dl(q):
while True:
url = q.get()
if isinstance(url, int):
time.sleep(1)
print(url)
else:
print("Fetching...")
print(f"{url} size is: {len(request.urlopen(request.Request(url)).read())}")
print("SUCCESS")
q.task_done()

q = Queue()
for i in range(slots):
t = Thread(target=dl, args=(q,))
t.daemon = True
t.start()

while True:
for n in range(4):
if n == 0:
q.put("https://example.com/")
elif n == 2:
q.put("https://example.net/")
else:
q.put(n)
q.join()
input("Enter to continue")
```

There are data fetching and 1s timers arranged in odd/even pattern. Success with slots = 1 (no parallel), success with slots = 2 if data fetch can complete before 1s timer.

Resorting to Requests same result.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running the supplied Python reproducer in ish with different slot counts, then compare the urllib and Requests cases described in the report. Done means parallel data fetching no longer hangs when timer and fetch tasks are interleaved.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.