`_abc._abc_subclasscheck` has very poor performance and (I think) a memory leak
未關閉
還沒有人認領這個 Issue。
extension-modules
performance
type-bug
- 主要語言
- Python
- 星號
- 77.2k
- 分支
- 36k
- PR 合併指標
- PR 指標待擷取
描述
Bug report
I'm been hunting a memory leak in pydantic and I keep seeing _abc_subclasscheck popping up in the memray flamegraph.
Locally I've seen memray report that _abc_subclasscheck is using 2.8GB of memory in some cases!
I can't get anything that bad in a minimal example, but I have the following:
from abc import ABCMeta
from datetime import datetime
class MyMetaclass(ABCMeta):
pass
class MyClass(metaclass=MyMetaclass):
pass
def main():
class Foobar(MyClass):
pass
assert issubclass(Foobar, MyClass)
assert not issubclass(int, MyClass)
assert not issubclass(str, MyClass)
assert not issubclass(datetime, MyClass)
t = type('A', (), {})
assert not issubclass(t, MyClass)
if __name__ == '__main__':
import os, psutil
process = psutil.Process(os.getpid())
mb = 1024 * 1024
last = 0
for i in range(5_000):
main()
# mem = process.memory_info().rss
# print(f'{i + 1:>4d} {mem / mb:8.2f}MB {(mem - last) / mb:+8.2f}MB | {"━" * int(mem / 8_000_000)}')
# last = mem
A few things to note:
- the commented out last few lines print current memory and change in memory over time
- I'm not sure exactly which
issubclasscalls are necessary, certainly I don't see a rise in memory with just the last one - some memory drops (presumably related to clearing the abc caches?) is happening, but overall memory is increasing
- The performance of
issubclasson an abc is very poor indeed - this script takes 14seconds, if i switchABCMetatotypeit takes 121ms!
Your environment
- Python 3.10.0 installed with venv
- Ubuntu 21.10
Linked PRs
- gh-131914
- gh-141171
- gh-144941
- gh-150540
貢獻指南
從這裡開始
- 先讀完整個 Issue,再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫,在一個分支上完成修改。
- 送出 Pull Request,並在描述裡引用這個 Issue 編號。
研究方向
首先執行提供的 Python 重現程式,並比較 ABCMeta 與 type,接著檢視連結的 PR gh-131914、gh-141171、gh-144941 和 gh-150540。完成的標準是識別並解決已回報的 issubclass 效能與記憶體增長行為,並提供來自重現程式的證據。
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- python
- 領域
- backend
- Issue 類型
- 缺陷
- 難度
- 5/5
- 預估耗時
- 一週以上
- 活躍度
- 停滯
- 描述清晰度
- 基本清楚
- 新手友好度
- 20/100