Improve handling of UTF-16-LE encoded .pth files

Feature or enhancement

I just finished an extended Windows bug hunt that I eventually tracked down to a .pth file being encoded in UTF-16-LE rather than an ASCII-compatible encoding.

I only figured it out by turning off frozen modules and hacking site.py to dump the output of .pth files as it tried to process them (at which point I checked the .pth file encoding in VSCode and sure enough, UTF-16-LE was down in the corner of the file window).

I hit the bug by porting a Linux shell script to Windows PowerShell, not thinking about the fact that | Out-File on Windows Powershell 5.1 defaults to UTF-16-LE (newer versions of PowerShell that aren't the ones baked into the OS default to UTF-8 without a BOM).

Given the inevitable presence of NULLs in a UTF-16-LE file, while there shouldn't be any in a UTF-8 or locale encoding file, it seems to me we should be able to handle such situations more gracefully (at the very least logging an error if NULL bytes are present in the file, but potentially even just making UTF-16-LE encoded .pth files straight up work by checking for NULL bytes, and using UTF-16-LE instead of UTF-8 and the locale encoding if we find one, or else trying UTF-16-LE before trying the locale encoding on Windows)

(This is somewhat related to #77102)

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

Linked PRs

• gh-119503
• gh-119508
• gh-119509

首先閱讀 site.py 對 .pth 檔案的處理方式，並使用 PowerShell 5.1 Out-File 產生的 UTF-16-LE 檔案重現 Windows 情況。在決定行為之前，檢視連結的 PR gh-119503、gh-119508 和 gh-119509。完成的標準是能妥善處理 UTF-16-LE .pth 檔案：要麼正常運作，要麼回報有用的錯誤。