On a relocatable Python install, setuptools should compute the C include path correctly

You can distribute CPython as a "relocatable" zip file, meaning, you just drop the zip file where-ever you want, and it works. For example, on Linux:

$ wget https://pybi.vorpus.org/cpython_unofficial-3.11.0-1-manylinux_2_17_x86_64.pybi
$ mkdir cpython-3.11
$ cd cpython-3.11
$ unzip ../cpython_unofficial-3.11.0-1-manylinux_2_17_x86_64.pybi
$ bin/python -c 'print("Hello world")'
Hello world

(You can also try this on macos or windows if you prefer.)

The resulting python knows where it is, where wheels should be installed, and all that; see sysconfig.get_paths():

$ bin/python -c 'import sysconfig, pprint; pprint.pprint(sysconfig.get_paths())'
{'data': '/tmp/cpython-3.11',
 'include': '/tmp/cpython-3.11/include/python3.11',
 'platinclude': '/tmp/cpython-3.11/include/python3.11',
 'platlib': '/tmp/cpython-3.11/lib/python3.11/site-packages',
 'platstdlib': '/tmp/cpython-3.11/lib/python3.11',
 'purelib': '/tmp/cpython-3.11/lib/python3.11/site-packages',
 'scripts': '/tmp/cpython-3.11/bin',
 'stdlib': '/tmp/cpython-3.11/lib/python3.11'}

Almost everything works great. But! When you use setuptools to compile a C extension, it uses the wrong include path:

$ bin/python -m ensurepip
$ bin/python -m pip install -U pip setuptools
$ mkdir foo
$ cd foo
$ echo '#include <Python.h>' > foo.c
$ cat <<EOF >setup.py
from setuptools import setup, Extension
setup(name="foo", ext_modules=[Extension("foo", sources=["foo.c"])])
EOF
$ ../bin/python setup.py build_ext
running build_ext
building 'foo' extension
creating build
creating build/temp.linux-x86_64-cpython-311
gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/pyinstall/include/python3.11 -c foo.c -o build/temp.linux-x86_64-cpython-311/foo.o
foo.c:1:10: fatal error: Python.h: No such file or directory
    1 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1

Notice that the gcc command line has -I/pyinstall/include/python3.11, which is a path on the machine where this Python interpreter was built, not the path where it's installed, or the path returned by sysconfig.get_paths().

I wasn't able to entirely follow where setuptools gets its include path from, but I think it's ultimately coming from distutils.sysconfig.get_python_inc, which invokes a number of strategies to find the include directory, but they're all based on static compile-time data, rather than the actual runtime location of the Python interpreter.

This is the only problem I've managed to find so far with these relocatable interpreters, so it would be nice if we could fix setuptools to use the correct include path :-).

Reproduce the failure with the relocatable interpreter and inspect distutils.sysconfig.get_python_inc, which the issue identifies as the likely source of the include path. Compare its result with sysconfig.get_paths() and trace how setuptools passes the path to the compiler. Done means the C extension build uses the installed interpreter's include directory and succeeds.