Possible memory leaking when combining session, threading and proxies
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 54.3k
- Forks
- 10.4k
- Avg merge
- 16h 43m
- Merged PRs (30d)
- 3
Description
I it helps I got error OSError: [Errno 24] Too many open files when running script, not sure if it is related with memory leak, I solved setting to 10000 ulimit -n 10000
Expected Result
RAM usage kept under reasonable limits
Actual Result
RAM usage doesn't stop growing
Reproduction Steps
I usually wouldn't be posting target website or the proxy credentials, but in this case I think they are needed for reproduce the bug.
import requests
from threading import Thread
from time import sleep
session = requests.Session()
from memory_profiler import profile
from random import randrange
finished = False
def get_proxy():
proxy = "http://lum-customer-hl_f53c879b-zone-static-session-" + str(randrange(999999)) + ":au2d3rzz8tut@zproxy.lum-superproxy.io:22225"
return {
"http": proxy,
"https": proxy
}
def make_request(url):
session.get(url, proxies=get_proxy())
def worker():
while True:
if finished: return
make_request("http://1000imagens.com/")
@profile
def main():
global finished
threads = []
for i in range(2):
t = Thread(target=worker)
t.start()
threads.append(t)
count = 0
while True:
sleep(1)
count += 1
if count == 300:
finished = True
return
main()
System Information
$ python3.9 -m requests.help
{
"chardet": {
"version": "3.0.4"
},
"cryptography": {
"version": ""
},
"idna": {
"version": "2.6"
},
"implementation": {
"name": "CPython",
"version": "3.9.1"
},
"platform": {
"release": "4.15.0-134-generic",
"system": "Linux"
},
"pyOpenSSL": {
"openssl_version": "",
"version": null
},
"requests": {
"version": "2.25.1"
},
"system_ssl": {
"version": "1010100f"
},
"urllib3": {
"version": "1.22"
},
"using_pyopenssl": false
}
# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.5 LTS
Release: 18.04
Codename: bionic
I tried with python versions 3.6, 3.8 and 3.9 and found no difference.
Output of memory_profiler
Line # Mem usage Increment Occurences Line Contents
============================================================
31 23.8 MiB 23.8 MiB 1 @profile
32 def main():
33 global finished
34 23.8 MiB 0.0 MiB 1 threads = []
35 23.8 MiB 0.0 MiB 3 for i in range(2):
36 23.8 MiB 0.0 MiB 2 t = Thread(target=worker)
37 23.8 MiB 0.0 MiB 2 t.start()
38 23.8 MiB 0.0 MiB 2 threads.append(t)
39
40 23.8 MiB 0.0 MiB 1 count = 0
41 while True:
42 547.1 MiB 523.2 MiB 300 sleep(1)
43 547.1 MiB 0.0 MiB 300 count += 1
44 547.1 MiB 0.0 MiB 300 if count == 300:
45 547.1 MiB 0.0 MiB 1 finished = True
46 547.1 MiB 0.0 MiB 1 return
After 5 minutes it eats +500MB ram. If I leave it running indefinitely it would consume all available ram and would be killed.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The report names no repository file or test; begin with the reproduction around Session.get(), worker(), and get_proxy(), using memory_profiler and the reported open-file error. Reproduce with the stated Python, Requests, and urllib3 versions, then trace the session and proxy request lifecycle. Done means identifying the cause and demonstrating bounded memory and file usage with a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100