`_abc._abc_subclasscheck` has very poor performance and (I think) a memory leak
未关闭
还没有人认领这个 Issue。
extension-modules
performance
type-bug
- 主要语言
- Python
- 星标
- 77.2k
- 派生
- 36k
- 平均合并
- 1 天 9 小时
- 30 天内合并 PR
- 558
描述
Bug report
I'm been hunting a memory leak in pydantic and I keep seeing _abc_subclasscheck popping up in the memray flamegraph.
Locally I've seen memray report that _abc_subclasscheck is using 2.8GB of memory in some cases!
I can't get anything that bad in a minimal example, but I have the following:
from abc import ABCMeta
from datetime import datetime
class MyMetaclass(ABCMeta):
pass
class MyClass(metaclass=MyMetaclass):
pass
def main():
class Foobar(MyClass):
pass
assert issubclass(Foobar, MyClass)
assert not issubclass(int, MyClass)
assert not issubclass(str, MyClass)
assert not issubclass(datetime, MyClass)
t = type('A', (), {})
assert not issubclass(t, MyClass)
if __name__ == '__main__':
import os, psutil
process = psutil.Process(os.getpid())
mb = 1024 * 1024
last = 0
for i in range(5_000):
main()
# mem = process.memory_info().rss
# print(f'{i + 1:>4d} {mem / mb:8.2f}MB {(mem - last) / mb:+8.2f}MB | {"━" * int(mem / 8_000_000)}')
# last = mem
A few things to note:
- the commented out last few lines print current memory and change in memory over time
- I'm not sure exactly which
issubclasscalls are necessary, certainly I don't see a rise in memory with just the last one - some memory drops (presumably related to clearing the abc caches?) is happening, but overall memory is increasing
- The performance of
issubclasson an abc is very poor indeed - this script takes 14seconds, if i switchABCMetatotypeit takes 121ms!
Your environment
- Python 3.10.0 installed with venv
- Ubuntu 21.10
Linked PRs
- gh-131914
- gh-141171
- gh-144941
- gh-150540
贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
调研方向
首先运行提供的 Python 复现程序,并将 ABCMeta 与 type 进行比较,然后查看相关的 PR gh-131914、gh-141171、gh-144941 和 gh-150540。完成的标准是识别并解决已报告的 issubclass 性能和内存增长行为,并提供来自复现程序的证据。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- python
- 领域
- backend
- Issue 类型
- 缺陷
- 难度
- 5/5
- 预计耗时
- 一周以上
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 20/100