Python hangs fetching data with urllib in parallel
- Dominant language
- C
- Stars
- 20.5k
- Forks
- 1.5k
- PR merge metrics
- No merged PRs in 30d
Description
```
import time
from urllib import request
from queue import Queue
from threading import Thread
slots = 4
def dl(q):
while True:
url = q.get()
if isinstance(url, int):
time.sleep(1)
print(url)
else:
print("Fetching...")
print(f"{url} size is: {len(request.urlopen(request.Request(url)).read())}")
print("SUCCESS")
q.task_done()
q = Queue()
for i in range(slots):
t = Thread(target=dl, args=(q,))
t.daemon = True
t.start()
while True:
for n in range(4):
if n == 0:
q.put("https://example.com/")
elif n == 2:
q.put("https://example.net/")
else:
q.put(n)
q.join()
input("Enter to continue")
```
There are data fetching and 1s timers arranged in odd/even pattern. Success with slots = 1 (no parallel), success with slots = 2 if data fetch can complete before 1s timer.
Resorting to Requests same result.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by running the supplied Python reproducer in ish with different slot counts, then compare the urllib and Requests cases described in the report. Done means parallel data fetching no longer hangs when timer and fetch tasks are interleaved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100