Implement PEP 793 – PyModExport

Feature or enhancement

Proposal:

See PEP 793. I'll copy the abstract for reference:

In this PEP, we propose a new entry point for C extension modules, by which one can define a module using an array of PyModuleDef_Slot structures without an enclosing PyModuleDef structure. This allows extension authors to avoid using a statically allocated PyObject, lifting the most common obstacle to making one compiled library file usable with both regular and free-threaded builds of CPython.

To make this viable, we also specify new module slot types to replace PyModuleDef’s fields, and to allow adding a token similar to the Py_tp_token used for type objects.

We also add an API for defining modules from slots dynamically.

The existing API (PyInit_*) is soft-deprecated. (That is: it will continue to work without warnings, and it’ll be fully documented and supported, but we plan to not add any new features to it.)

• Initial implementation (#140556)
• Docs:
  • Reference docs (#141197)
    • Soft-deprecate PyInit_* (#141197)
    • Better explanation of module state (#141197)
    • Give an example for tokens (#141197)
  • What's New
    • a notice for DYNLOADFILE users? (edit: This shouldn't have changed/broken more than usual.)
  • Porting guide HOWTO
    • Document that the export hook runs without GIL in FT builds (see #141780) (#151661)
  • Tutorial update (#142314)
  • Mentions in PyImport_AppendInittab & friends (#151661)
  • Make sure references to new API are linked:
    • PyMod_Execin PyImport_CreateModuleFromInitfunc docs is linked (#141197)
    • PyMODEXPORT_FUNC in blurb for #141672
    • blurb for #141780
• xxlimited.c update (#142827)
  • Remove/replace _Py_OPAQUE_PYOBJECT in xxlimited.c (by 3.15.0 beta)

Optional stretch goals:

• Remove the PyInit function from Lib/test/test_cext/extension.c (https://github.com/python/cpython/pull/141511)
• Speed & size improvements
  • importdl: load the DLL once
  • bool for the GIL flag (#141519)
• No longer require spec.name in PyModule_FromSlotsAndSpec
• No longer require static allocation for Py_mod_methods
• Add a PyModule_GetStateWithToken() function

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

Linked PRs

• gh-140556
• gh-141056
• gh-141197
• gh-141511
• gh-141519
• gh-142583
• gh-142827
• gh-144340
• gh-149397
• gh-149785
• gh-151661
• gh-152064
• gh-157493
• gh-157494
• gh-157539
• gh-157560

See also:

• gh-141672
• gh-142314

Review the PEP 793 proposal and the remaining unchecked stretch goals, especially Lib/test/test_cext/extension.c and the importdl item. Start by checking the linked implementation and documentation work, then identify one specific remaining goal and its existing tests or entry points; done means that goal is implemented and covered by the relevant documentation or tests.