Improve performance by using reference stealing methods
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 77.2k
- Forks
- 35.9k
- PR merge metrics
- PR metrics pending
Description
Feature or enhancement
Proposal:
We can improve performance of cpython by using stealing methods (_PyList_AppendTakeRef, _PyDict_SetItem_Take2, PyTuple_FromPairSteal etc.) instead of the normal ones. Using stealing methods we typically avoid an incref/decref pair and sometimes error handling is simplified. We should only make changes where it makes a performance impact, the code is not more complex and the modules or methods affected are performance critical.
A list of possible cases (there might be more).
| Location | Module / context | Helper |
|---|---|---|
Modules/_csv.c:688 |
csv reader, per field | _PyList_AppendTakeRef |
Modules/pyexpat.c:472 |
pyexpat StartElement, per attribute | _PyDict_SetItem_Take2 |
Modules/_elementtree.c:3351 |
ElementTree C accel attrib dict | _PyDict_SetItem_Take2 |
Modules/_io/bytesio.c:577 |
io.BytesIO.readlines | _PyList_AppendTakeRef |
Modules/_io/iobase.c:756 |
io.IOBase.readlines (generic) | _PyList_AppendTakeRef |
Objects/frameobject.c:639 & 658 |
frame f_locals / vars() / tracing |
_PyList_AppendTakeRef |
Modules/_pickle.c:5138 |
pickle Pickler memo copy | _PyDict_SetItem_Take2 |
Modules/_io/textio.c:2337 & 2343 |
io.TextIOWrapper readline (multi-chunk) | _PyList_AppendTakeRef |
Modules/_asynciomodule.c:1213 |
asyncio future done-callback compaction | _PyList_AppendTakeRef |
Update: Most cases already have an open PR, for new ones please open a separate issue.
Linked PRs
- gh-150943
- gh-150945
- gh-150995
- gh-151002
- gh-151062
- gh-151209
- gh-151608
- gh-153013
- gh-154860
- gh-155171
- gh-155304
- gh-155922
- gh-155940
- gh-155957
- gh-156055
- gh-156199
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the listed C locations, including Modules/_csv.c, Modules/pyexpat.c, Modules/_elementtree.c, and the _io, Objects, _pickle, and _asynciomodule files. First check the linked PRs to see which cases are already covered; any remaining work should be split into a separate, scoped issue with relevant performance tests. Done means a selected performance-critical case is changed without added complexity and its behavior and performance are verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100