astral-sh / astral-sh/python-build-standalone

Investigate x86_64 build versions (microarchitectures) and whether they can be simplified

Open
#736 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
4.4k
Forks
314
Avg merge
1d 9h
Merged PRs (30d)
27

Description

We're currently spending a lot of CI time/cost on building for x86_64, x86_64-v2, x86_64-v3, and x86_64-v4. uv isn't using these yet (astral-sh/uv#9788). I'm not sure who is but it certainly makes our list of release binaries much noisier.

I'd like to understand
* whether these are actually worth building (with benchmarks) - I would guess it's more important that packages like numpy are optimized for your CPU than that Python and its dependencies are.
* whether we can use runtime detection instead of build-time - the [GCC `target_clones` extension](https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-target_005fclones-function-attribute) might be useful here, or we could do the dispatching manually. In either case this would be patches to the relevant upstream projects, unless they already support this.

Ideally we would do a single, compatible, fast-enough build - either by deciding that v0 is fast enough or by doing runtime dispatching.

Related questions to answer:
* Is the Python JIT detecting the runtime µarch and making full use of it? Should it?
* I think we ended up compiling out some of the SIMD stuff in HACL, can we turn it back on? (Or can we turn off HACL entirely, since we ship OpenSSL?)
* I know glibc is good at providing multiple optimized memcpy etc. implementations and detecting the best one for thje runtime µarch. Does musl do that too, and if so, is that working properly in our static musl builds?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.