jhao104 / jhao104/proxy_pool

内存泄露问题

Open
#522 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
23.7k
Forks
5.4k
PR merge metrics
No merged PRs in 30d

Description

代理进程一直跑着没管,今天有个其它服务被oom了,查了一下oom记录,发现proxy_pool schedule进程内存占用达16G。。。
我直接好家伙。。。
查了半天,大概率是多线程下requests库引起的,详情可见requests库的#4601。
试了以下几个解决方案:
1. 在涉及requests发起请求后添加session.close()和res.close()。没用
2. 手动del session & del res。没用
3. 手动gc.collect()。没用
4. 因为进程结束后内存自动释放给os,而线程结束后相关内存由解释器管理(当然也就导致了内存泄露),所以将任务逻辑使用subprocess触发。解决

解决效果:schedule进程内存开销在32M左右,抓取和检测两个任务进程的内存开销都保持在100M以内。

辛苦作者看下能不能复现。

我的requests环境
```
$ python -m requests.help
{
"chardet": {
"version": "3.0.4"
},
"cryptography": {
"version": ""
},
"idna": {
"version": "2.7"
},
"implementation": {
"name": "CPython",
"version": "3.6.5"
},
"platform": {
"release": "2.6.32_1-19-0-0",
"system": "Linux"
},
"pyOpenSSL": {
"openssl_version": "",
"version": null
},
"requests": {
"version": "2.20.0"
},
"system_ssl": {
"version": "1000109f"
},
"urllib3": {
"version": "1.24.3"
},
"using_pyopenssl": false
}
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.