aspect-build / aspect-build/rules_py
[FR]: Sophisticated sdist_build support
- Dominant language
- Starlark
- Stars
- 145
- Forks
- 97
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 71
Description
### What is the current behavior?
Today the `sdist_build` support can be reasonably accused of being a placeholder. We need a rule which goes from a source distribution archive to an installable platform-specific way in the most generic possible manner.
Achieving this in the simple cases especially of legacy `-none-any` packages which just don't ship prebuilds is trivial, and about what's currently implemented. But the entire point of the uv project was to achieve generalized support for C-extensions such as PyO3 and cython which require the use of at a minimum a C toolchain and likely non-Python build time dependencies in order to produce hermetic build results. Today there's a hat tip in this direction in the form of the `sdist_native_build` rule, which uses execution constraints to allow for RBE crossbuilds, but there's no coherent mechanism in the current uv extension for native sdist builds to be autoconfigured, or for appropriate toolchains to be identified and injected as part of build orchestration.
### Describe the feature
Long time Bazel ecosystem participants will recognize that this is just the latest incarnation of what seems to be the ever ongoing struggle between Bazel's phasing model which demands static dependencies and the outside world where dynamic dependencies are replete. The conventional solution to this problem is exemplified by the Gazelle and `rules_go` stack, which tries to cheat the phasing model by using a prebuilt tool during the repository phase to perform code and dependency analysis which can drive `BUILD.bazel` generation according to dependencies which it is intractable to analyze statically.
https://pypackaging-native.github.io/key-issues/native-dependencies/ has a useful writeup of some of the challenges in the Python ecosystem. The Astral folks have made mention of interest in efforts to standardize a native dependency story but as of today no such really exists. Pixi is a partial solution to this problem, https://github.com/astral-sh/uv/pull/14735 and https://github.com/astral-sh/uv/issues/2252 are I believe germane.
For more modern packages, there are conventions for specifying as requirements packages which must be installed in order to perform a source build. I do not believe that such build time transitive dependencies are included in UV's lock solutions at present. For older setuptools based packages interpretation of `setup.py` files is required. This is made worse by legacy `setup.py` systems which perform custom `cmake` and such calls.
An idea I've batted around before and which is in many ways an extension of the current `annotations.toml` pattern is to create a non-standardized rules_py specific "database" format noting the build time dependencies for key Python ecosystem packages, as a possible input to partial solutions to this rats nest of issues.
I highlight all these issues not to propose that we solve them comprehensively but to impress that there is at present to _reasonable_ let alone _comprehensive_ solution to these problems. The best I think we can technically achieve is to adjust the design of the ruleset so that we provide an sdist build configuration interface which is modular to a substantial degree precisely so that we do not have to be in the business of owning a comprehensive solution to this problem space; users can bring their own and if a more comprehensive ecosystem solution does become available hopefully we can integrate it without fuss.
---
The sketch I would propose is that we combine the new PBS toolchain support and the prebuilt tool pattern to allow for us to invoke a "simple" Python script with a PBS sourced interpreter during the repository phase of configuring `sdist_build` repos. Following the example of `rules_go` we would replace at least most `sdist_build` buildfile generation with a Python script whose responsibility would be to generate a conventional build target, using whatever rules rulesets and dependencies it may require. This would allow the satisfaction of @zbarsky-openai's ask that we have general support for sdist builds consuming toolchains such as cc and make, by allowing users to bring a configuration script which will do so appropriately without having to weigh down our baseline `sdist_build` rule with dependencies on toolchains which are irrelevant to the majority of wheel builds and perhaps worse inadequate to the configuration needs of those wheels which do have advanced builds.
Contributor guide
Research direction
Start by reading the current sdist_build and sdist_native_build implementations, along with the annotations.toml pattern and PBS toolchain support mentioned in the issue. Done would mean a modular configuration interface that lets users provide repository-phase build configuration and toolchain dependencies without requiring a comprehensive native-dependency solution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100