[BUG] Race condition due to reuse of build folder when building extensions in parallel

setuptools version

setuptools=70.0.0

Python version

Python 3.12

OS

RHEL 9

Additional environment information

No response

Description

Having multiple extensions in ext_modules being compiled in parallel with pip install --config-setting='--build-option=build_ext' --config-setting='--build-option=-j24' ... causes issues because they share the same build folder set at https://github.com/pypa/setuptools/blob/5a13876673a41e3cd21d4d6e587f53d0fb4fd8e5/setuptools/_distutils/command/build.py#L111-L112

So when they write anything into that folder they may overwrite or delete files used by other extensions

Expected behavior

Building extensions in parallel has the same result as building serially

How to Reproduce

• Add multiple extensions to ext_modules that use e.g. the same source files but compile them e.g. with different flags.
• Install with the parallel flag set

This was observed with DeepSpeed so you could try installing that from source with "enough" cores but it might be too much to setup

Output

Output varies and it might succeed depending on the amount of parallel actions going on and the available CPU cores. So it may sometimes succeed or fail seemingly randomly

Start with setuptools/_distutils/command/build.py at the linked lines, then reproduce with multiple ext_modules that share source files but use different compilation flags and a parallel build. Compare parallel and serial results; done means parallel extension builds produce the same result without overwriting or deleting files used by another extension.