bazel-contrib / bazel-contrib/rules_python
Allow using environment for configuring toolchains and pypi with bzlmod
- Ngôn ngữ chính
- Starlark
- Star
- 688
- Fork
- 721
- Merge trung bình
- 15 giờ 7 phút
- Pull request đã merge (30 ngày)
- 76
Mô tả
Over in [jax](https://github.com/jax-ml/jax) (and some related projects, like xla), they're currently using workspace and have a pretty bespoke way of configuring their toolchains and pip settings.
They use environment variables to specify the python version, url, sha, and threading. Those generate some repos with the values, which eventually decide/feed into the `python_repository/python_register_toolchains/pip_parse` rules. The net effect is they have one python version for the whole build, but are able to change it without modifying workspace. CI jobs and users can then set the values to change what python is used (both for the toolchain and for pip). Thus, they're able to have something resembling multi-version support. The basic logic of their WORKSPACE is something like:
```
load("env.bzl", "env")
env() # reads env vars, generates '@env'
load("@env//...", "HERMETIC_...")
if url:
python_register_toolchains(name="python", TOOL_VERSIONS=)
else:
python_register_toolchains(name="python", >)
pip_parse(version=, interpreter=@python//:interpreter)
```
Something this env var setup allows that our builtin multi-version setup doesn't is allowing the user to _easily_ specify an alternative python runtime. e.g. they simply do `HERMETIC_PYTHON_VERSION=3.15, HERMETIC_PYTHON_URL=file:///cpython-3.15.tar.gz`, and then its used for everything, including by pip_parse.
I think there's two basic needs this is trying to serve:
1.. Allowing easily overriding the runtime. This allows custom building Python (at head, with santizers, etc) and using it.
2. pip_parse can be sensitive to the runtime used. This design helps ensure the right interpreter is used. In particular, if a freethreading interpreter is used.
For (1), local toolchain rules should be able to handle this, mostly. The problem I see is several bzlmod APIs want a string literal for python version, but with such a toolchain, we don't know the version until its run.
For (2), pip.parse can use python_interpreter_target to point to a local runtime. However, (1) python_version is required, which we don't know, and (2) pip.parse is particular about duplicate calls.
Sketching a MODULE.bazel, I came up with this:
```
local_runtime = use_repo_rule(...)
local_runtime(name="local_runtime", path="python3")
local_toolchain = use_repo_rule(...)
local_toolchain(name="local_toolchains", repos=["local_runtime"], TCW=)
register_toolchains("@local_toolchains//...")
pip = use_extension(...)
pip.parse(
python_version = ???
python_interpreter_target = "@local_runtime//:python3")
requirements = "//:requirements-local.txt",
config_settings = ???
)
# Run
export PATH=$PATH:/cpython-src/build/python3.15
bazel build --@//:py=local //:foo
```
The python_version and config_settings part for pip.parse is unclear.
Maybe add `python_version_target="@local_runtime//:version.txt"` ? If we could get rid of the python_version attribute entirely that might be better. Is it actually required (during bzlmod/repo phase) if the interpreter target is given explicitly?
I'm not sure what config_settings would be for pip.parse. Maybe just match what is set on the toolchain?
Some misc improvments to local toolchain that might help:
* Allow local toolchains to get python from a particular envvar. (modifying PATH seems invasive, potentially expensive)
* Generate a bzl file with the detected python version. There's some various contexts where a loading-phase string is needed of the python version (py_wheel, py_binary.python_version, among others). These should probably be updated to accept a label for the python version, where feasible.
Hướng dẫn đóng góp
Hướng nghiên cứu
Bắt đầu với bản phác thảo MODULE.bazel và toolchain cục bộ, cùng các API python_repository/python_register_toolchains và pip.parse được mô tả trong issue; so sánh chúng với luồng WORKSPACE/env.bzl. Theo dõi nơi python_version, python_interpreter_target và config_settings được sử dụng, sau đó xác định các test cho runtime được môi trường lựa chọn và cho pip sử dụng cùng một trình thông dịch. Được xem là hoàn tất khi luồng bzlmod được hỗ trợ và hành vi bắt buộc của nó được xác định, đồng thời được bao phủ bởi các test.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- python
- Lĩnh vực
- build-system
- Loại issue
- Tính năng
- Độ khó
- 5/5
- Thời gian dự kiến
- Hơn một tuần
- Mức độ hoạt động
- Đình trệ
- Độ rõ ràng
- Cần làm rõ
- Mức phù hợp với người mới
- 25/100