psf / psf/requests

Possible memory leaking when combining session, threading and proxies

Open
#5,726 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
54.3k
Forks
10.4k
Avg merge
16h 43m
Merged PRs (30d)
3

Description

I it helps I got error OSError: [Errno 24] Too many open files when running script, not sure if it is related with memory leak, I solved setting to 10000 ulimit -n 10000

Expected Result

RAM usage kept under reasonable limits

Actual Result

RAM usage doesn't stop growing

Reproduction Steps

I usually wouldn't be posting target website or the proxy credentials, but in this case I think they are needed for reproduce the bug.

import requests
from threading import Thread
from time import sleep

session = requests.Session()
from memory_profiler import profile
from random import randrange
finished = False


def get_proxy():
    proxy = "http://lum-customer-hl_f53c879b-zone-static-session-" + str(randrange(999999)) + ":au2d3rzz8tut@zproxy.lum-superproxy.io:22225"
    return {
        "http": proxy,
        "https": proxy
    }


def make_request(url):
    session.get(url, proxies=get_proxy())

def worker():
    while True:
        if finished: return
        make_request("http://1000imagens.com/")


@profile
def main():
    global finished
    threads = []
    for i in range(2):
        t = Thread(target=worker)
        t.start()
        threads.append(t)

    count = 0
    while True:
        sleep(1)
        count += 1
        if count == 300:
            finished = True
            return

main()

System Information

$ python3.9 -m requests.help
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.9.1"
  },
  "platform": {
    "release": "4.15.0-134-generic",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.25.1"
  },
  "system_ssl": {
    "version": "1010100f"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": false
}
# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.5 LTS
Release:        18.04
Codename:       bionic

I tried with python versions 3.6, 3.8 and 3.9 and found no difference.

Output of memory_profiler

Line #    Mem usage    Increment  Occurences   Line Contents
============================================================
    31     23.8 MiB     23.8 MiB           1   @profile
    32                                         def main():
    33                                             global finished
    34     23.8 MiB      0.0 MiB           1       threads = []
    35     23.8 MiB      0.0 MiB           3       for i in range(2):
    36     23.8 MiB      0.0 MiB           2           t = Thread(target=worker)
    37     23.8 MiB      0.0 MiB           2           t.start()
    38     23.8 MiB      0.0 MiB           2           threads.append(t)
    39
    40     23.8 MiB      0.0 MiB           1       count = 0
    41                                             while True:
    42    547.1 MiB    523.2 MiB         300           sleep(1)
    43    547.1 MiB      0.0 MiB         300           count += 1
    44    547.1 MiB      0.0 MiB         300           if count == 300:
    45    547.1 MiB      0.0 MiB           1               finished = True
    46    547.1 MiB      0.0 MiB           1               return

After 5 minutes it eats +500MB ram. If I leave it running indefinitely it would consume all available ram and would be killed.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The report names no repository file or test; begin with the reproduction around Session.get(), worker(), and get_proxy(), using memory_profiler and the reported open-file error. Reproduce with the stated Python, Requests, and urllib3 versions, then trace the session and proxy request lifecycle. Done means identifying the cause and demonstrating bounded memory and file usage with a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.