[BUG] Bad MANIFEST.in exclude can globally break sdist when setuptools_scm is installed

setuptools version

62.4.0
setuptools-scm 6.4.2

Python version

3.10.5

OS

Arch Linux, reproducible on fedora

Additional environment information

No response

Description

I discovered an interesting setuptools foot-gun that took me a few hours to figure out.

1. Using wildcard global excludes in MANIFEST.in, it is possible to create installable but broken packages by excluding certain metadata files. Minimal example:

$ tar -taf normal-sdist.tar.gz
fnord-23.0.42/
fnord-23.0.42/MANIFEST.in
fnord-23.0.42/PKG-INFO
fnord-23.0.42/fnord.egg-info/
fnord-23.0.42/fnord.egg-info/PKG-INFO
fnord-23.0.42/fnord.egg-info/SOURCES.txt
fnord-23.0.42/fnord.egg-info/dependency_links.txt
fnord-23.0.42/fnord.egg-info/top_level.txt
fnord-23.0.42/setup.cfg
fnord-23.0.42/setup.py
$ tar -taf broken-sdist.tar.gz
fnord-23.0.42/
fnord-23.0.42/PKG-INFO
fnord-23.0.42/fnord.egg-info/
fnord-23.0.42/fnord.egg-info/SOURCES.txt
fnord-23.0.42/setup.cfg

Notice how the following files are missing:

fnord-23.0.42/fnord.egg-info/PKG-INFO
fnord-23.0.42/fnord.egg-info/dependency_links.txt
fnord-23.0.42/fnord.egg-info/top_level.txt

2. setuptools'es sdist command trips up with a TypeError: expected string or bytes-like object when it finds such a package installed in the user's ~/.local/lib/python*/site-packages (stack trace below).

Expected behavior

• setuptools should refuse to build a package that is missing essential metadata files.
• I'm not sure what the intended semantics of MANIFEST.in are, but matching its includes/excludes against package-internal metadata files that are autogenerated during the package build seems counter-intuitive to me. If this is required behavior, it would be good if it were hidden behind a switch.
• sdist should not break when encountering an invalid package on disk
• When sdist due to a bug errors out while parsing package meta from site-packages, it should print the name and path of the offending package along with the error to give the user some hint on how to fix the issue.

How to Reproduce

In a fedora container, run:

python3 -m ensurepip
python3 -m pip install -UI 'setuptools'
python3 -m pip install -UI 'setuptools_scm'
python3 -m pip install -UI 'build'

mkdir /tmp/testpkg
cd /tmp/testpkg
cat <<EOS > setup.py
from setuptools import setup, find_packages

setup(
    name='fnord',
    version='23.0.42',
    python_modules=['fnord']
)
EOS
cat <<EOS > MANIFEST.in
global-exclude *
EOS
python3 setup.py install --user
python3 -m build --sdist -n

Output

Future calls to setup.py sdist or python3 -m build --sdist -n will fail with a TypeError: expected string or bytes-like object, even in setup.py of completely unrelated projects under the same account:

[root@f5db61a4677a /]# python3 -m ensurepip
WARNING: Running pip install with root privileges is generally not a good idea. Try `python3 -m pip install --user` instead.
Looking in links: /tmp/tmpt3_2fo1d
Processing /tmp/tmpt3_2fo1d/setuptools-49.1.3-py3-none-any.whl
Processing /tmp/tmpt3_2fo1d/pip-20.2.2-py2.py3-none-any.whl
Installing collected packages: setuptools, pip
Successfully installed pip-20.2.2 setuptools-49.1.3
[root@f5db61a4677a /]# python3 -m pip install -UI 'setuptools'
WARNING: Running pip install with root privileges is generally not a good idea. Try `python3 -m pip install --user` instead.
Collecting setuptools
  Downloading setuptools-62.4.0-py3-none-any.whl (1.2 MB)
     |████████████████████████████████| 1.2 MB 41.0 MB/s 
Installing collected packages: setuptools
Successfully installed setuptools-62.4.0
WARNING: You are using pip version 20.2.2; however, version 22.1.2 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
[root@f5db61a4677a /]# python3 -m pip install -UI 'setuptools_scm'
WARNING: Running pip install with root privileges is generally not a good idea. Try `python3 -m pip install --user` instead.
Collecting setuptools_scm
  Downloading setuptools_scm-6.4.2-py3-none-any.whl (37 kB)
Collecting packaging>=20.0
  Downloading packaging-21.3-py3-none-any.whl (40 kB)
     |████████████████████████████████| 40 kB 14.4 MB/s 
Collecting setuptools
  Using cached setuptools-62.4.0-py3-none-any.whl (1.2 MB)
Collecting tomli>=1.0.0
  Downloading tomli-2.0.1-py3-none-any.whl (12 kB)
Collecting pyparsing!=3.0.5,>=2.0.2
  Downloading pyparsing-3.0.9-py3-none-any.whl (98 kB)
     |████████████████████████████████| 98 kB 22.0 MB/s 
Installing collected packages: pyparsing, packaging, setuptools, tomli, setuptools-scm
Successfully installed packaging-21.3 pyparsing-3.0.9 setuptools-62.4.0 setuptools-scm-6.4.2 tomli-2.0.1
WARNING: You are using pip version 20.2.2; however, version 22.1.2 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
[root@f5db61a4677a /]# python3 -m pip install -UI 'build'
WARNING: Running pip install with root privileges is generally not a good idea. Try `python3 -m pip install --user` instead.
Collecting build
  Downloading build-0.8.0-py3-none-any.whl (17 kB)
Collecting tomli>=1.0.0; python_version < "3.11"
  Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
Collecting pep517>=0.9.1
  Downloading pep517-0.12.0-py2.py3-none-any.whl (19 kB)
Collecting packaging>=19.0
  Using cached packaging-21.3-py3-none-any.whl (40 kB)
Collecting pyparsing!=3.0.5,>=2.0.2
  Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Installing collected packages: tomli, pep517, pyparsing, packaging, build
Successfully installed build-0.8.0 packaging-21.3 pep517-0.12.0 pyparsing-3.0.9 tomli-2.0.1
WARNING: You are using pip version 20.2.2; however, version 22.1.2 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
[root@f5db61a4677a /]# 
[root@f5db61a4677a /]# mkdir /tmp/testpkg
[root@f5db61a4677a /]# cd /tmp/testpkg
[root@f5db61a4677a testpkg]# cat <<EOS > setup.py
> from setuptools import setup, find_packages
> 
> setup(
>     name='fnord',
>     version='23.0.42',
>     python_modules=['fnord']
> )
> EOS
[root@f5db61a4677a testpkg]# cat <<EOS > MANIFEST.in
> global-exclude *
> EOS
[root@f5db61a4677a testpkg]# python3 setup.py install --user
/usr/local/lib/python3.9/site-packages/setuptools/_distutils/dist.py:257: UserWarning: Unknown distribution option: 'python_modules'
  warnings.warn(msg)
running install
/usr/local/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/usr/local/lib/python3.9/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
running bdist_egg
running egg_info
creating fnord.egg-info
writing fnord.egg-info/PKG-INFO
writing dependency_links to fnord.egg-info/dependency_links.txt
writing top-level names to fnord.egg-info/top_level.txt
writing manifest file 'fnord.egg-info/SOURCES.txt'
reading manifest file 'fnord.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'fnord.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
warning: install_lib: 'build/lib' does not exist -- no Python modules to install

creating build
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating dist
creating 'dist/fnord-23.0.42-py3.9.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing fnord-23.0.42-py3.9.egg
Copying fnord-23.0.42-py3.9.egg to /root/.local/lib/python3.9/site-packages
Adding fnord 23.0.42 to easy-install.pth file

Installed /root/.local/lib/python3.9/site-packages/fnord-23.0.42-py3.9.egg
Processing dependencies for fnord==23.0.42
Finished processing dependencies for fnord==23.0.42
[root@f5db61a4677a testpkg]# python3 -m build --sdist -n
* Getting dependencies for sdist...
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pep517/in_process/_in_process.py", line 363, in <module>
    main()
  File "/usr/local/lib/python3.9/site-packages/pep517/in_process/_in_process.py", line 345, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/usr/local/lib/python3.9/site-packages/pep517/in_process/_in_process.py", line 297, in get_requires_for_build_sdist
    return hook(config_settings)
  File "/usr/local/lib/python3.9/site-packages/setuptools/build_meta.py", line 181, in get_requires_for_build_sdist
    return self._get_build_requires(config_settings, requirements=[])
  File "/usr/local/lib/python3.9/site-packages/setuptools/build_meta.py", line 159, in _get_build_requires
    self.run_setup()
  File "/usr/local/lib/python3.9/site-packages/setuptools/build_meta.py", line 281, in run_setup
    super(_BuildMetaLegacyBackend,
  File "/usr/local/lib/python3.9/site-packages/setuptools/build_meta.py", line 174, in run_setup
    exec(compile(code, __file__, 'exec'), locals())
  File "setup.py", line 3, in <module>
    setup(
  File "/usr/local/lib/python3.9/site-packages/setuptools/__init__.py", line 86, in setup
    _install_setup_requires(attrs)
  File "/usr/local/lib/python3.9/site-packages/setuptools/__init__.py", line 75, in _install_setup_requires
    dist = MinimalDistribution(attrs)
  File "/usr/local/lib/python3.9/site-packages/setuptools/__init__.py", line 57, in __init__
    super().__init__(filtered)
  File "/usr/local/lib/python3.9/site-packages/setuptools/dist.py", line 475, in __init__
    for ep in metadata.entry_points(group='distutils.setup_keywords'):
  File "/usr/local/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 999, in entry_points
    return SelectableGroups.load(eps).select(**params)
  File "/usr/local/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 449, in load
    ordered = sorted(eps, key=by_group)
  File "/usr/local/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 996, in <genexpr>
    eps = itertools.chain.from_iterable(
  File "/usr/local/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/_itertools.py", line 16, in unique_everseen
    k = key(element)
  File "/usr/local/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 931, in _normalized_name
    return self._name_from_stem(stem) or super()._normalized_name
  File "/usr/local/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 600, in _normalized_name
    return Prepared.normalize(self.name)
  File "/usr/local/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 855, in normalize
    return re.sub(r"[-_.]+", "-", name).lower().replace('-', '_')
  File "/usr/lib64/python3.9/re.py", line 210, in sub
    return _compile(pattern, flags).sub(repl, string, count)
TypeError: expected string or bytes-like object

ERROR Backend subprocess exited when trying to invoke get_requires_for_build_sdist

Reproduce the failure with the provided setup.py, MANIFEST.in, installation, and python3 -m build --sdist -n commands. Start by tracing setuptools/build_meta.py and setuptools/init.py into the vendored importlib_metadata entry-point parsing shown in the traceback. Done means invalid installed metadata no longer breaks unrelated sdist builds, and failures identify the offending package and path.