bazel-contrib / bazel-contrib/rules_python

Allow using environment for configuring toolchains and pypi with bzlmod

オープン
#3,293 コメント 5 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Starlark
スター
688
フォーク
721
平均マージ
15時間 7分
マージ済み PR(30日)
76

説明

Over in [jax](https://github.com/jax-ml/jax) (and some related projects, like xla), they're currently using workspace and have a pretty bespoke way of configuring their toolchains and pip settings.

They use environment variables to specify the python version, url, sha, and threading. Those generate some repos with the values, which eventually decide/feed into the `python_repository/python_register_toolchains/pip_parse` rules. The net effect is they have one python version for the whole build, but are able to change it without modifying workspace. CI jobs and users can then set the values to change what python is used (both for the toolchain and for pip). Thus, they're able to have something resembling multi-version support. The basic logic of their WORKSPACE is something like:

```
load("env.bzl", "env")
env() # reads env vars, generates '@env'
load("@env//...", "HERMETIC_...")
if url:
python_register_toolchains(name="python", TOOL_VERSIONS=)
else:
python_register_toolchains(name="python", >)
pip_parse(version=, interpreter=@python//:interpreter)
```

Something this env var setup allows that our builtin multi-version setup doesn't is allowing the user to _easily_ specify an alternative python runtime. e.g. they simply do `HERMETIC_PYTHON_VERSION=3.15, HERMETIC_PYTHON_URL=file:///cpython-3.15.tar.gz`, and then its used for everything, including by pip_parse.

I think there's two basic needs this is trying to serve:
1.. Allowing easily overriding the runtime. This allows custom building Python (at head, with santizers, etc) and using it.
2. pip_parse can be sensitive to the runtime used. This design helps ensure the right interpreter is used. In particular, if a freethreading interpreter is used.

For (1), local toolchain rules should be able to handle this, mostly. The problem I see is several bzlmod APIs want a string literal for python version, but with such a toolchain, we don't know the version until its run.

For (2), pip.parse can use python_interpreter_target to point to a local runtime. However, (1) python_version is required, which we don't know, and (2) pip.parse is particular about duplicate calls.

Sketching a MODULE.bazel, I came up with this:

```
local_runtime = use_repo_rule(...)
local_runtime(name="local_runtime", path="python3")
local_toolchain = use_repo_rule(...)
local_toolchain(name="local_toolchains", repos=["local_runtime"], TCW=)
register_toolchains("@local_toolchains//...")

pip = use_extension(...)

pip.parse(
python_version = ???
python_interpreter_target = "@local_runtime//:python3")
requirements = "//:requirements-local.txt",
config_settings = ???
)

# Run
export PATH=$PATH:/cpython-src/build/python3.15
bazel build --@//:py=local //:foo
```

The python_version and config_settings part for pip.parse is unclear.

Maybe add `python_version_target="@local_runtime//:version.txt"` ? If we could get rid of the python_version attribute entirely that might be better. Is it actually required (during bzlmod/repo phase) if the interpreter target is given explicitly?

I'm not sure what config_settings would be for pip.parse. Maybe just match what is set on the toolchain?

Some misc improvments to local toolchain that might help:
* Allow local toolchains to get python from a particular envvar. (modifying PATH seems invasive, potentially expensive)
* Generate a bzl file with the detected python version. There's some various contexts where a loading-phase string is needed of the python version (py_wheel, py_binary.python_version, among others). These should probably be updated to accept a label for the python version, where feasible.

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

Start with the MODULE.bazel sketch and the local toolchain, python_repository/python_register_toolchains, and pip.parse APIs described in the issue; compare them with the WORKSPACE/env.bzl flow. Trace where python_version, python_interpreter_target, and config_settings are consumed, then identify tests for environment-selected runtimes and pip using the same interpreter. Done means the supported bzlmod flow and its required behavior are defined and covered.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
python
領域
build-system
issue の種類
機能追加
難易度
5/5
見積もり時間
1週間以上
活発さ
停滞
明瞭さ
説明が足りない
初心者へのやさしさ
25/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。