python / python/cpython

`PyType_GetModuleByDef` is really slow.

Open
#119,663 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

interpreter-core performance
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR merge metrics
PR metrics pending

Description

PyType_GetModuleByDef does a lot of work for what was a single memory read prior to sub-interpreters.

For example, https://github.com/python/cpython/issues/114682.

We should provide a more efficient way to get to the module state from the module def.

AFAICT, the function to get the module state is a pure function of the interpreter and the module def.
Since the module def already has a m_index field that is described as "The module's index into its interpreter's modules_by_index cache.", then a Py_GetModuleFromDef() function could be as simple as:

inline PyObject *
Py_GetModuleFromDef(PyModuleDef *def)
{
     return PyInterpreterState_Get()->modules_table[def->m_base.m_index];
}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with PyType_GetModuleByDef, PyModuleDef.m_index, and the interpreter's modules_by_index cache named in the issue. Compare the current lookup path with the proposed module-from-definition API and verify that behavior remains correct per interpreter while addressing the reported overhead.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, python
Domain
backend-api-design, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.