84.0.0: compilers obtained via `setuptools._distutils.ccompiler` raise a `CompileError` that `setuptools.errors.CompileError` no longer matches

setuptools version

84.0.0 (regression from 83.0.0)

Python

3.12 and 3.13 (reproduced on both)

OS

macOS (arm64) and Debian trixie (linux/aarch64 container)

Description

A compiler obtained through setuptools._distutils.ccompiler.new_compiler() now raises
setuptools._distutils.compilers.C.errors.CompileError, which is a different class
object from setuptools.errors.CompileError. Handlers written as
except CompileError: against the public setuptools.errors name therefore no longer
catch compile failures from such a compiler.

To be clear about scope: the purely public path is unaffected. A compiler from
distutils.ccompiler.new_compiler() (i.e. via the distutils hack), and the compiler
used by setuptools' own build_ext, both still raise the class that
setuptools.errors.CompileError resolves to. Under 84.0.0:

>>> from setuptools.errors import CompileError
>>> from distutils.ccompiler import new_compiler          # public path
>>> type(new_compiler()).__module__
'distutils.compilers.C.unix'                              # -> raises the matching class

>>> from setuptools._distutils.ccompiler import new_compiler   # private path
>>> type(new_compiler()).__module__
'setuptools._distutils.compilers.C.unix'                  # -> raises a different class

The two module trees (distutils.* and setuptools._distutils.*) refer to the same
files, but Python imports them as separate module objects, so their CompileError
classes are not identical. That duplication exists in 83.0.0 too — what changed is that
83.0.0 forced the concrete compiler class to be loaded under the distutils.* name
regardless of how new_compiler was reached. In compilers/C/base.py, 83.0.0 had:

module_name = "distutils." + module_name
__import__(module_name)
module = sys.modules[module_name]
klass = vars(module)[class_name]

84.0.0 replaces that lookup with get_compilers()[compiler], which resolves the class
through the package's own relative imports. Reached via setuptools._distutils, that
now yields setuptools._distutils.compilers.C.unix.UnixCCompiler, whose CompileError
comes from the private tree. The hardcoded "distutils." prefix was, in effect, keeping
the two trees' exception identities unified for every caller.

Diagnostic (abridged output — the compiler's own stderr and some logging warnings are
omitted; full script below):

=== setuptools 83.0.0 ===
caught class : distutils.compilers.C.errors CompileError
raised class : distutils.compilers.C.errors CompileError
caught is raised: True
isinstance match: True

=== setuptools 84.0.0 ===
caught class : distutils.compilers.C.errors CompileError
raised class : setuptools._distutils.compilers.C.errors CompileError
caught is raised: False
isinstance match: False

Real-world impact

setuptools_dso — the build helper used by the EPICS Python packages (pvxslibs,
epicscorelibs, softioc; the PEP 517 backend is still setuptools.build_meta) —
mixes the two import paths, and has done so deliberately for years:

# setuptools_dso/compiler.py
from setuptools.errors import ExecError, CompileError

# Seems like the easiest fix for now as these don't appear
# to be exposed directly through setuptools yet
# See https://github.com/pypa/setuptools/issues/2806
from setuptools._distutils.ccompiler import (new_compiler as _new_compiler, ...)

It reaches for the private module precisely because new_compiler is not exported
publicly (#2806). It then probes the toolchain by test-compiling snippets and treats a
compile failure as "feature absent" via except (ExecError, CompileError). Under
84.0.0 that handler stops matching, so the first probe for a header that is legitimately
absent escapes as an uncaught exception and the wheel build fails:

  File ".../setuptools_dso/probe.py", line 154, in check_include
    return self.check_includes([header], **kws)
  ...
  File ".../setuptools/_distutils/compilers/C/unix.py", line 329, in _compile
    raise CompileError(msg)
setuptools._distutils.compilers.C.errors.CompileError: Command '['gcc', ...
    '-c', '/tmp/.../try_compile.c', ...]' returned non-zero exit status 1.
ERROR: Failed building wheel for pvxslibs

This only bites where the package must be built from sdist. Those projects publish no
aarch64 wheels (pvxslibs 1.5.2 ships macOS universal2, manylinux x86_64 and win_amd64
only), so pip install of them on linux/aarch64 — which in practice means container
image builds on arm64 — always builds from source, and now fails. The same build
succeeds when the build environment is constrained to setuptools<84.

How to Reproduce

Requires a working C compiler on PATH (the probe actually compiles a snippet).

$ python -m venv v84 && v84/bin/pip install setuptools==84.0.0 setuptools_dso==2.12.3
$ v84/bin/python repro.py
setuptools: 84.0.0
caught class : distutils.compilers.C.errors CompileError
raised class : setuptools._distutils.compilers.C.errors CompileError
caught is raised: False
isinstance match: False

Repeat with setuptools==83.0.0 for the contrast (caught is raised: True).

repro.py:

import setuptools
import setuptools_dso.probe as probe_mod

print("setuptools:", setuptools.__version__)
print("caught class :", probe_mod.CompileError.__module__, probe_mod.CompileError.__qualname__)

p = probe_mod.ProbeToolchain()
try:
    p.compile("#include <definitely_not_a_real_header_xyz.h>")
except Exception as e:
    cls = type(e)
    print("raised class :", cls.__module__, cls.__qualname__)
    print("caught is raised:", cls is probe_mod.CompileError)
    print("isinstance match:", isinstance(e, probe_mod.CompileError))

The compiler will print its own "no such file" error to stderr in between; that is
expected and is the failure being probed for.

Without setuptools_dso, the two-line snippet in the Description section shows the same
split using only setuptools.

End-to-end: pip install --no-binary pvxslibs pvxslibs fails with 84.0.0 in the build
environment and succeeds when constrained to setuptools<84.

Expected behavior

Either of these would resolve it:

• setuptools._distutils.compilers.C.errors.CompileError and the CompileError that
  setuptools.errors re-exports are the same object, so that which of the two
  equivalent module trees a caller imported through does not change exception identity;
  or
• new_compiler (and friends) get a supported public export, so downstream code has no
  reason to import from setuptools._distutils in the first place (#2806).

I appreciate that setuptools._distutils is private and that a project importing from
it has limited standing. Filing anyway because the duplicate-class-identity hazard is
easy to trip over generally, and because the previous behavior masked it for every
caller, so the change lands as a silent runtime break rather than an import error.

Workaround

Constrain build environments to setuptools<84 with a PIP_CONSTRAINT file —
the environment variable (unlike --constraint) does propagate into pip's isolated
PEP 517 build environments:

$ echo 'setuptools<84' > constraints.txt
$ PIP_CONSTRAINT=$PWD/constraints.txt pip install pvxslibs

Start by reading compilers/C/base.py, especially the get_compilers() lookup, then run the issue's repro.py with setuptools 84.0.0 and 83.0.0. Trace setuptools._distutils.ccompiler.new_compiler() and the setuptools.errors.CompileError export; done means compiler failures are caught by the public exception identity or a supported public compiler entry point removes the private import need.