Add a Check for Extensions that Support Running in Subinterpreters

Any given extension module may or may not support being loaded in multiple interpreters. This primarily depends on whether or not the extension use any process-global state. When imported in multiple interpreters at the same time, a module that does not isolate its state may lead to problems.

Global state (which must be isolated) includes:

• Python objects the extension uses (including internally)
• other data it uses directly
• indirect state, via linked libraries

If an extension properly implements multi-phase init (PEP 489) then it probably does support use in multiple interpreters (but not necessarily under per-interpreter GIL). In fact, PEP 489 proscribes this.

It would be useful to add a check to the ExtensionFileLoader to this effect.

Proposal

• if an extension is imported in a sub-interpreter and it does not implement multi-phase init then we should raise ImportError
• this should be configurable on a per-interpreter basis
• the default is debatable
• we may want to allow multi-phase init extension to opt in to multiple interpreter support
• (there are some additional minor considerations if we factor in a per-interpreter GIL)

Configuration

Given a _PyInterpreterConfig (see gh-98608), we would add something like _PyInterpreterConfig.check_multi_interp_extensions to indicate that the interpreter created via _PyInterpreterNewFromConfig() should do the check or not.

For backward compatibility, we'd use the following defaults:

• "false" for the main interpreter
• (maybe) "false" for the legacy Py_NewInterpreter()
• "true" for "isolated" interpreters (see the _xxsubinterpreters module, PEPs 554, 684)

For Py_NewInterpreter() it may make sense to default to "true", to avoid causing folks headaches (see https://github.com/pyca/cryptography/issues/2299). However, we'll probably want to stick with "false" under the assumption that most extensions work well enough in multiple interpreters to allow us to preserve compatibility.

A Module Def Slot

Per PEP 684 discussions, there is some uncertainty about how confident extension authors may be about how isolated their extensions really are. If this is a genuine concern then we should consider also adding a PEP 489 module def slot, e.g. Py_mod_subinterpreters, to allow extension authors to opt in to supporting multiple interpreters. It would probably also make sense to offer a slot to opt out of multiple interpreter support, e.g. Py_mod_no_subinterpreters, for when support is more wide-spread.

Again, this only makes sense if we aren't confident that multi-phase init support is an adequate indicator.

Per-Interpreter GIL (PEP 684)

There are additional static variables, e.g. temporary buffers, that factor in to extension "isolation" if you drop the thread safety provided by the GIL. This is real concern relative to PEP 684, where we would make the GIL per-interpreter. The difference between isolation is relatively small, but enough to warrant consideration.

Most notably, a per-interpreter GIL would strengthen the value of a dedicated module def slot.

• PR: gh-99039

• PR: gh-99040