import performance: large codebases that use a lot of `attrs` classes start taking a long time to load
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 32/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- python
- Domain
- performance
Research direction
Begin in attr._make and trace the generated-source, compile(), and eval() paths while measuring import time for large codebases with many attrs classes. The work is done when rigorous measurements establish the impact and a caching approach is agreed that preserves transparency, debug-ability, and PyPy compatibility.
Written by the indexing model from the issue text.
Description
I need to do some more rigorous analysis of this, so my level of confidence here is "strong hunch", but: using attrs extensively, across a large codebase, can contribute non-trivially to Python's scourge of slow load times.
Many things in Python are slow: calling functions, importing modules, and so on. But one of the slowest things is reading and parsing Python code. This is why Python compiles and caches .pyc files - it's a substantial optimization to startup time.
Using attrs, unfortunately, partially undoes this optimization, because many of the methods in attr._make:
- generate a bunch of source code
compile()it, theneval()the bytecode they just compiled.
Now, they do this for a good reason. Transparency and debug-ability are great, and more than once being able to "see through" the attrs validator stack while debugging has been legitimately useful. So I wouldn't want to propose a technique that makes a substantially different tradeoff. Not to mention that friendliness to PyPy's JIT is achieved via this mechanism, and definitely wouldn't want to trade that off. But could we possibly piggyback on Python's native caching technique here, and cache both the .pyc output and the source code, in the __pycache__ directory or thereabouts?
- Dominant language
- Python
- Stars
- 5.8k
- Forks
- 480
- Avg merge
- 2h 15m
- Merged PRs (30d)
- 2
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from python-attrs/attrs
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
python-attrs/attrs#1620 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 55/100
python-attrs/attrs#1596 · 2 comments · 1 reaction ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
python-attrs/attrs#1549 · 3 comments ·
-
Difficulty 4/5 3-5 days Newbie friendliness 38/100
python-attrs/attrs#1543 · 2 comments ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
python-attrs/attrs#1532 ·
All issues in python-attrs/attrs
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
zostera/django-bootstrap4#894 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
use-agent-os/agent-os#3276 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
zephyrproject-rtos/zephyr#119726 ·
-
area/auth bug comp/agent P3 platform/discord type/security
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
NousResearch/hermes-agent#117848 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
zilliztech/memsearch#759 ·