`_abc._abc_subclasscheck` has very poor performance and (I think) a memory leak
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 77.2k
- Forks
- 36k
- PR merge metrics
- PR metrics pending
Description
Bug report
I'm been hunting a memory leak in pydantic and I keep seeing _abc_subclasscheck popping up in the memray flamegraph.
Locally I've seen memray report that _abc_subclasscheck is using 2.8GB of memory in some cases!
I can't get anything that bad in a minimal example, but I have the following:
from abc import ABCMeta
from datetime import datetime
class MyMetaclass(ABCMeta):
pass
class MyClass(metaclass=MyMetaclass):
pass
def main():
class Foobar(MyClass):
pass
assert issubclass(Foobar, MyClass)
assert not issubclass(int, MyClass)
assert not issubclass(str, MyClass)
assert not issubclass(datetime, MyClass)
t = type('A', (), {})
assert not issubclass(t, MyClass)
if __name__ == '__main__':
import os, psutil
process = psutil.Process(os.getpid())
mb = 1024 * 1024
last = 0
for i in range(5_000):
main()
# mem = process.memory_info().rss
# print(f'{i + 1:>4d} {mem / mb:8.2f}MB {(mem - last) / mb:+8.2f}MB | {"━" * int(mem / 8_000_000)}')
# last = mem
A few things to note:
- the commented out last few lines print current memory and change in memory over time
- I'm not sure exactly which
issubclasscalls are necessary, certainly I don't see a rise in memory with just the last one - some memory drops (presumably related to clearing the abc caches?) is happening, but overall memory is increasing
- The performance of
issubclasson an abc is very poor indeed - this script takes 14seconds, if i switchABCMetatotypeit takes 121ms!
Your environment
- Python 3.10.0 installed with venv
- Ubuntu 21.10
Linked PRs
- gh-131914
- gh-141171
- gh-144941
- gh-150540
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the supplied Python reproducer and comparing ABCMeta with type, then review the linked PRs gh-131914, gh-141171, gh-144941, and gh-150540. Done means identifying and resolving the reported issubclass performance and memory-growth behavior, with evidence from the reproducer.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 20/100