[FR] Use UTF-8 for `MANIFEST.in`

What's the problem this feature will solve?

I’m trying to make my package work consistently across Linux and Windows. My MANIFEST.in file contains Unicode characters. What encoding MANIFEST.in is decoded with depends on the user’s locale settings. Most Linux distros default to UTF-8. Windows defaults to a non-Unicode encoding (in fact, UTF-8 locales are a rather recent feature in Windows. My Windows 10 system still labels them as being in beta). What this means is that my package successfully builds on most Linux systems and fails to build with a UnicodeDecodeError on most Windows systems.

Describe the solution you'd like

Make it so that MANIFEST.in files are always decoded as UTF-8. This idea is similar to #1702.

Alternative Solutions

At the moment, I tell users who want to build the package to either make their locale use UTF-8 or enable UTF-8 mode. This isn’t ideal because it requires extra effort on the users part. Plus, I’m the one who created the file; I would know best what encoding it uses, not the OS.

Another solution would be to allow package authors’ to configure the encoding used by MANIFEST.in. A similar option for configuring setup.cfg’s encoding was added but was removed in favor of always using UTF-8. Based on that, I’m assuming that this project would prefer to just require UTF-8 instead of allowing the encoding to be configured.

Finally, the easiest solution would be for me to only use ASCII characters in my MANIFEST.in file. Theoretically, this would still fail on systems with a locale encoding that isn’t compatible with ASCII, but I’m not sure if any of those systems would be able to run modern Python.

Additional context

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct

Start by tracing how setuptools reads MANIFEST.in, using the UTF-8 handling change referenced in issue #1702 as context. Confirm the behavior on a MANIFEST.in containing Unicode characters and add a regression test; done means the file is decoded as UTF-8 consistently on Linux and Windows.