Move vendored modules into a `vendor` directory

Feature or enhancement

Proposal:

In CPython, we have some vendored libraries namely libmpdec, hacl, and expat. Those libraries are meant to be clone of their upstream (think of them as git submodules) and many times have I been exluding them from code search as they usually have non-CPython code involved.

We have a plan to remove vendored libmpdec (#115119) and we're almost here. In the long term, we could also try to unvendor expat (probably not hacl as it's used to implement hash functions fallbacks when OpenSSL is not present).

Affected modules:

• libmpdec
• expat
• _hacl

Some advantages:

• Easy exclusion of vendored modules from code search.
• Easy addition and deletion of vendored modules.
• We don't expect redistributors to edit vendored code (especially not hacl).
• Hopefully less work than the Python/Programs split
• Could be using git submodules (though this requires an internet connection, so maybe not)

Some inconvenients:

• Quite painful refactoring since it would affect existing PRs.
• Quite painful refactoring since it would affect distributors editing those files.

We can start with some modules that should be kept untouched such as HACL* sources and progressively move the others to reduce the work and conflicts. It doesn't need to happen in one go (for instance, we may well ignore the libmpdec case if we manage to make it unvendored before). I don't think we have much open PRs with expat (by the way, we could have a refresh script for expat to ease maintenance like https://github.com/python/cpython/issues/126623).

Now, the question is how this could affect downstream redistributors. I'm asking first on Github since I don't know whether they are active on Discourse or not. If everyone tells "it's fine", then I'll ask on Discourse to see if there are more redistributors that could be concerned.

cc

• @zooba (Windows)
• @mhsmith (Android) ✅
• @doko42 / @stefanor (Ubuntu/Debian)
• @mitya57 (Debian)
• @hroncok / @vstinner (Fedora/RHEL) ✅
• @mgorny (Gentoo) ✅
• @danigm / @mcepl (openSUSE) ✅
• @freakboy3742 / @ned-deily (macOS / iOS) ✅
• @zanieb (relocatable builds)
• @hugovk / @erlend-aasland (CPython)
• @mweinelt (NixOS) ✅
• @gpshead / @ncoghlan / @brettcannon / @zware (involved through Discord messages or reactions)

I don't know how moving mimalloc related stuff would affect the free-threaded build in particular, so I'm also going to ask @kumaraditya303 and @colesbury about it. EDIT: Turns out it's a no go for mimalloc as there is some CPython dedicated stuff, so we can put it out of the list (see https://github.com/python/cpython/issues/129222#issuecomment-2610160032).

For the hacl includes, I can take care of it.

Beginne damit, zu prüfen, wie die betroffenen vendored modules—libmpdec, expat und _hacl—in CPython organisiert sind und auf sie verwiesen wird, und bewerte anschließend die in der Diskussion angesprochenen Anliegen der nachgelagerten Redistributoren. Als erledigt gilt die Aufgabe, wenn ein Verzeichnislayout und ein Migrationsplan vereinbart sind, die Build-Nutzer, Distributoren und Module berücksichtigen, die möglicherweise vendored bleiben.