Upstream-supported cross-compilation

Currently, we are reliant on parsing _sysconfigdata files to get the configuration we need for cross-compilation.

While better than what we had before (parsing C header files, which was extremely broken), it is reliant on internals of the Python interpreter so we cannot expect this to be working permanently. As the number of users of PyO3 grows the importance of relying only on stable interfaces will significantly increase.

See below opinion from @mattip (PyPy maintainer) in https://github.com/PyO3/pyo3/issues/1487#issuecomment-855364623_

There is no standard to _sysconfigdata (with the underline), no consensus of how to ship multiple files, and no documentation of what values are expected to be in it.

There is also no standard environment variable _PYTHON_SYSCONFIGDATA_NAME, the discussion in the issue with the commit pointed to in the cpython code apparently happened at a sprint so there is no documentation of the thought that went into to the change and what is expected to be in a shipped python installation. So it is very difficult for PyPy to be compliant with this idea.

See issues PyO3/maturin#559 and the corresponding PyPy issue https://foss.heptapod.net/pypy/pypy/-/issues/3483. I am trying to reconstruct how this is supposed to be used in cross-compiling for PyPy (which does not use configure), and how, for instance, the use of these internal interfaces works on Ubuntu or a standard manylinux docker image where I fail to see any additional _sysconfigdata* files other than the ones for the host interpreter. Is there a design document for cross-compilation?

FWIW I agree with this; the current system was built out of necessity to find something which worked with CPython to support immediate needs of users trying to do this.

It would be very much worth engaging with the upstream Python foundation to work on a documented way (a PEP?) to retrieve information necessary for cross-compilation. The official solution for configuration of builds (afaik) is to run a Python interpreter and use the sysconfig module, but that doesn't work in cross-compilation where the target Python interpreter can't be executed locally.

crossenv is a third-party solution for cross-compilation which creates a local interpreter with a heavily-patched sysconfig module suitable for cross-compiling.

One could imagine, for example, a json document which shipped with every Python distribution (i.e. very similar to _sysconfigdata) which would be a known public specification. Or perhaps the sysconfig module in the stdlib could be extended to support querying for cross-compilation information.

If anyone has time to champion this, we should start reaching out at https://discuss.python.org/c/ideas

I'd be willing to do this myself at some point in the future, however I can't guarantee when.

Start by reviewing the current reliance on Python _sysconfigdata files and the documented sysconfig behavior for cross-compilation. Read the linked Python discussions and compare the crossenv approach. Done would require an upstream-supported, documented way to obtain target configuration without executing the target interpreter, rather than a change confined to PyO3.