Building and linking C extensions in a post-distutils world

With the removal of distutils from 3.12 alphas, I've recently taken a new look at the use of distutils within https://mesonbuild.com in the hopes that we were finally unblocked and could migrate to sysconfig.

• install schemes turn out to finally be viable since 3.10.3, when the deb_system scheme patch on Debian operating systems for distutils was patched into sysconfig. This hard blocker is gone, thank G-d.
• linking to libpython itself may or may not be an issue, originally I thought it was, but now I think it may not be.

How can a C-extension supporting PEP 517 build backend know the best way to compile and link?

python <3.12

Meson has some code: https://github.com/mesonbuild/meson/blob/master/mesonbuild/modules/python.py#L338-L407

Here's the interesting bit:

def links_against_libpython():
    from distutils.core import Distribution, Extension
    cmd = Distribution().get_command_obj('build_ext')
    cmd.ensure_finalized()
    return bool(cmd.get_libraries(Extension('dummy', [])))

python >=3.8

In bpo-36721, bpo-21536 etc the above code changes from returning True to False, on many Unix platforms.

New additional methods suitable for getting C extension arguments are available. Well, mostly suitable.

• sysconfig.get_config_var('LIBPYTHON')

this is configured into:

• pkg-config --cflags --libs python3
• python-config --embed

There's a couple problems with this:

• on Cygwin and Android, LIBPYTHON and thus pkg-config is hardcoded to link to libpython, but distutils only does so when Py_ENABLE_SHARED
• none of it works on Windows, which since it is built with PCBuild, doesn't have Makefile config vars (and also / consequently? distributes neither of the latter two)
• pkg-config may not always be installed, so we want a fallback for that
• python-config has tons of CPython build-time junk so we cannot use it

...

It feels uncomfortably like there's still way too much undocumented magic here.

@vstinner, I assume the Cygwin/Android inconsistency is probably a configure.ac bug that should behave the same way distutils does/did? I can provide a patch to fix it but would like confirmation of which side to fix.

@FFY00, I think in the long term, sysconfig should expose a function that returns cflags / libs required to build an extension (and works on Windows without _generate_posix_vars, and doesn't include lots of -O3 -pipe -fstack-protector-strong -fdiagnostics-color=always and more -- i.e. works like pkg-config, not like python-config). Even without the question of "whether to link to libpython", there's a 140-line class for figuring out what the name of the library is, which directory to find it in, and what the include directory is too. Of course, this is specific to the case where pkg-config is not available, and most of it is for the Windows case.

Linked PRs

• gh-100356
• gh-100967