[BUG] Setuptools fails in editable install mode when a nonstandard build location is used

setuptools version

setuptools==69.0.2

Python version

Python 3.11.6

OS

MacOS Sonoma (14.1.2)

Additional environment information

No response

Description

I am working with a project that uses Bazel to build a C++ extension via a setuptools.Extension and subclassing the build_ext.build_ext class.

In the current setup (which worked for a long time until about a month ago in CI), Bazel is symlinking its build directories directly into build_ext.build_temp.

Unfortunately, during the course of an editable install via pip (calling python -m pip install -e .), setuptools tries to copy the built extension from the build_ext.build_lib directory, which it evidently expects to contain the extensions. Here's the relevant source:

https://github.com/pypa/setuptools/blob/dd5f15a600a071ba00859e84fa497a9d1a25f521/setuptools/command/build_ext.py#L99-L101

Later in the build_ext.copy_extensions_to_source() API, these extensions are copied even if the assumed extension path (given by build_ext._get_inplace_equivalent()) does not exist.

https://github.com/pypa/setuptools/blob/dd5f15a600a071ba00859e84fa497a9d1a25f521/setuptools/command/build_ext.py#L105-L112

(L111 will still execute if the extension does not exist in the expected location, but is required).

This is exactly the case if, for example, a build tool with its own hermetic output directory (like Bazel) is used, and thus the install will fail.

This was a head scratcher for me until debugging a bit more, since python -m pip install . (fixed, not editable) worked just fine.

I am not sure whether this really qualifies as a bug - my hunch is that if I set an alternative build directory on the build_ext or the setuptools.Extension in the correct location, it might have continued working. But the sudden breakage, the editable vs. non-editable discrepancy and the customization docs not mentioning it made me go ahead with this report.

Is there a canonical way to correctly wire output directories of build tools in build_ext subclasses?

Expected behavior

It continues working.

How to Reproduce

NB: Having Bazel installed is a prerequisite for running the repro below.

git clone https://github.com/google/benchmark.git
cd benchmark
git checkout HEAD~5 # bug is already fixed on main, so rewind a couple of commits
python3 -m venv venv --system-site-packages --upgrade-deps # should fetch latest setuptools
source venv/bin/activate
python -m pip install -e .

Output

      Traceback (most recent call last):
        File "/private/var/folders/rw/mj164vm16k10x_byww6my1s00000gn/T/pip-build-env-rimjgckl/overlay/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 156, in run
          self._create_wheel_file(bdist_wheel)
        File "/private/var/folders/rw/mj164vm16k10x_byww6my1s00000gn/T/pip-build-env-rimjgckl/overlay/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 345, in _create_wheel_file
          files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/rw/mj164vm16k10x_byww6my1s00000gn/T/pip-build-env-rimjgckl/overlay/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 268, in _run_build_commands
          self._run_build_subcommands()
        File "/private/var/folders/rw/mj164vm16k10x_byww6my1s00000gn/T/pip-build-env-rimjgckl/overlay/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 295, in _run_build_subcommands
          self.run_command(name)
        File "/private/var/folders/rw/mj164vm16k10x_byww6my1s00000gn/T/pip-build-env-rimjgckl/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/private/var/folders/rw/mj164vm16k10x_byww6my1s00000gn/T/pip-build-env-rimjgckl/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 963, in run_command
          super().run_command(command)
        File "/private/var/folders/rw/mj164vm16k10x_byww6my1s00000gn/T/pip-build-env-rimjgckl/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "<string>", line 56, in run
        File "/private/var/folders/rw/mj164vm16k10x_byww6my1s00000gn/T/pip-build-env-rimjgckl/overlay/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 91, in run
          self.copy_extensions_to_source()
        File "/private/var/folders/rw/mj164vm16k10x_byww6my1s00000gn/T/pip-build-env-rimjgckl/overlay/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 112, in copy_extensions_to_source
          self.copy_file(regular_file, inplace_file, level=self.verbose)
        File "/private/var/folders/rw/mj164vm16k10x_byww6my1s00000gn/T/pip-build-env-rimjgckl/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 350, in copy_file
          return file_util.copy_file(
                 ^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/rw/mj164vm16k10x_byww6my1s00000gn/T/pip-build-env-rimjgckl/overlay/lib/python3.11/site-packages/setuptools/_distutils/file_util.py", line 115, in copy_file
          raise DistutilsFileError(
      distutils.errors.DistutilsFileError: can't copy '/var/folders/rw/mj164vm16k10x_byww6my1s00000gn/T/tmpfg283hkc.build-lib/google_benchmark/_benchmark.cpython-311-darwin.so': doesn't exist or not a regular file
      /private/var/folders/rw/mj164vm16k10x_byww6my1s00000gn/T/pip-build-env-rimjgckl/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py:988: _DebuggingTips: Problem in editable installation.

Start with setuptools/command/build_ext.py around the referenced run and copy_extensions_to_source lines, then inspect editable_wheel.py and reproduce the failure using the google/benchmark checkout and the provided editable-install command. Done means the Bazel-backed editable install no longer attempts to copy a missing extension from the assumed build_lib path, while fixed installs remain working.