Tracking Issue: Sub-Interpreter Support

Tracks the development and state of supporting sub-interpreters in PyO3.

This issue really only tracks progress, for discussing everything else, feel free to join over here: https://github.com/Aequitosh/pyo3/discussions/1

Summary

As of 13.09.2023

PyO3 currently doesn't support sub-interpreters, which will lead to an ImportError being raised if a module using PyO3 is initialized more than once per interpreter process. As stated in https://github.com/PyO3/pyo3/pull/2523, this is necessary in order to prevent soundness holes (as in, prevent things that use PyO3 from randomly breaking, having nasty undefined behaviour, etc.).

Even though this prevents soundness holes on the one hand, it can lead to modules / applications using a sub-interpreter model to "break" in certain situations. For examples, see https://github.com/pyca/cryptography/issues/9016 and https://github.com/bazaah/aur-ceph/issues/20.

Implementing sub-interpreter support isn't straightforward and requires quite a substantial redesign of PyO3's API. This issue shall track this redesign and provide as much relevant information as possible for all that wish to contribute.

Goals

Adapted from https://github.com/PyO3/pyo3/issues/576#issuecomment-1713975913, as of 13.09.2023.

Mid-Term

• Rework synchronization primitives to not rely on the GIL. See https://github.com/PyO3/pyo3/pull/2885
  • Develop transition plan so that existing users can migrate their code without enormous amounts of work
  • Remove static data from PyO3's implementation, either move things to PyModule_GetState (preferred) or PyInterpreterState_GetDict (alternative)
• Allow extension authors to use unsafe in order to opt in to sub-interpreter support - it is their responsibility to guarantee to not store Py<T> in any static data.
• Document all conditions that extension authors' modules need to satisfy so that they may be used within sub-interpreters

Long-Term

Possibly remove the need for extension authors to audit their own code once we're confident enough.

Tasks

TBA - might them here (or some other place) once more concrete pieces of work have been identified.

Relevant Issues & Interesting Reads

Listing relevant things here. Some things might already be linked above, but it's nevertheless nice to have everything in one place.

• Initial discussion regarding sub-interpreter support:
  #576
• PR regarding nogil Python support, contains lots of additional information:
  https://github.com/PyO3/pyo3/pull/2885
• Discussion regarding making Python's C-API more friendly for Rust; linking to comment what would need to happen in PyO3 internally:
  https://github.com/PyO3/pyo3/discussions/2346#discussioncomment-2911159
• cryptography issue regarding sub-interpreters in PyO3:
  https://github.com/pyca/cryptography/issues/9016
• aur-ceph maintainer's issue regarding Ceph's sub-interpreter model, and why Ceph Dashboard breaks:
  https://github.com/bazaah/aur-ceph/issues/20
• Idea by @GoldsteinE - using a ghostcell-ish pattern:
  https://github.com/PyO3/pyo3/issues/576#issuecomment-1713999916

This is a tracking issue for a substantial PyO3 API redesign, not a self-contained task, and it names no files or tests. Start by reading issue #576 and PR #2885, then review the listed synchronization, static-data, opt-in, and documentation goals; the issue does not define a concrete done condition.