ENH: Use multi-phase initialisation (PEP 489) for extension modules

Proposed new feature or change:

This is intended to be a tracking issue for a series of PRs I will shortly submit.

Multi-phase initialisation was introduced in PEP 489 for Python 3.5. It replaces the previous 'single-phase' mechanism by splitting the module creation process into creation and execution phases. Quoting from the documentation:

Extension modules created this way behave more like Python modules: the initialization is split between the creation phase, when the module object is created, and the execution phase, when it is populated. The distinction is similar to the __new__() and __init__() methods of classes.

This is also the stable mechanism to declare compatibility with both free-threading and subinterpreters (though note that this PR does not attempt to add support for subinterpreters).

A common point of confusion/conflation is multi-phase initialisation and module isolation. As noted on the Python Discourse forum, "Implementing extension module isolation is a very good thing, but if a module uses static types or (C) global variables then the effort to isolate becomes non-trivial.". This issue only seeks to implement multi-phase initialisation, generally following the guide that Eric set out in that thread:

1. move the content of the module’s init function to a corresponding “module exec” function (dropping the call to create the module object)
2. set the module def’s m_slots field to an array with:

• a Py_mod_exec slot, set to the new module exec function
• a Py_mod_multiple_interpreters slot, set to Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED
• to try free-threading, a Py_mod_gil slot, set to Py_MOD_GIL_NOT_USED

3. set def.m_size to 0 (if negative)
4. update the module init function to only return PyModuleDef_Init(def); for the corresponding module def

cc @ngoldbaum

xref:

• https://github.com/numpy/numpy/issues/26157#issuecomment-2079486111
• https://github.com/numpy/numpy/issues/28271#issuecomment-2634857326
• https://github.com/numpy/numpy/issues/24755
• https://github.com/numpy/numpy/issues/11925#issuecomment-421082106
• https://github.com/numpy/numpy/issues/665

A

PRs:

• #29022
• #29023
• #29024
• #29025
• #29026
• #29027
• #29028
• #29029
• #29030
• #29031
• TODO: simd_dispatch
• TODO: f2py

Note: substantial discussion on initial multi-phase changes is contained in GH-29030 ('Convert umath_linalg to multi-phase init').

A

This is a tracking issue for a series of changes rather than a standalone task. Start by reviewing PRs #29022–#29031 and the substantial discussion in GH-29030, then check the remaining TODO items for simd_dispatch and f2py. The work is complete when the planned extension-module conversions are addressed across those follow-up tasks.