bazel-contrib / bazel-contrib/rules_python
launching python subprocess can fail: `rules_python==1.7.0` + `bootstrap_impl=system_python`
- 主要言語
- Starlark
- スター
- 688
- フォーク
- 721
- 平均マージ
- 15時間 7分
- マージ済み PR(30日)
- 76
説明
# 🐞 bug report
### Affected Rule
`py_binary() / py_test()` that invokes python as subprocess. When using `--@rules_python//python/config_settings:bootstrap_impl=system_python` (default)
`--@rules_python//python/config_settings:bootstrap_impl=script` does not reproduce this issue.
### Is this a regression?
**Did this behavior use to work in the previous version?** Yes
### Description
I have had python executables where they launched a python subprocess as part of them. A basic one for example looked like
```py
# Python code
import sys
# do some preprocess
subprocess.check_output([sys.executable, "-m", "datamodel_code_generator", ...
# do some post process
```
```py
# BUILD code
py_binary(
name = "pydantic_generator",
deps = ["@pypi//:datamodel_code_generator", ...]
)
```
With `rules_python==1.7.0` I get an error that no module named `datamodel_code_generator` found.
## 🔬 Minimal Reproduction
https://github.com/shayanhoshyari/issue-reports/tree/main/rules_python/subprocess_call
## 🔥 Exception or Error
/private/var/tmp/_bazel_hoshyari/5bf493163e144814d5d8244a8cb0233d/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/test.runfiles/rules_python++python+python_3_13_aarch64-apple-darwin/bin/python3: No module named datamodel_code_generator
## 🌍 Your Environment
**Operating System:**
```
Reproduces on both ubuntu (tried 22.04) MacOS (tried Sonoma)
```
**Output of `bazel version`:**
```
bazel 8.2.1
```
**Rules_python version:**
```
1.7.0
```
**Anything else relevant?**
Seems this is due to https://github.com/bazel-contrib/rules_python/commit/b8e32c454a1158cd78ce4ecaef809b99bef4e5da
1. `1.6.3` and older used `PYTHONPATH` so it propagated in subprocess calls.
2. `1.7.0` starts using the combination
1. [bootstrap stage 1](https://github.com/bazel-contrib/rules_python/blob/411b937484b7a690fad2bacf2d03fec60bed765b/python/private/python_bootstrap_template.txt) that calls
2. [bootstrap stage 2](https://github.com/bazel-contrib/rules_python/blob/main/python/private/stage2_bootstrap_template.py) that adds `iii` to `site.addsitepackage`, then runs main entrypiont with `runpy`.
3. [siteinit.py](https://github.com/bazel-contrib/rules_python/blob/main/python/private/site_init_template.py) that adds the new imports.
The issue is that adding folders via `site.addsitepackage` do not propagate via `subprocess.call`. So when the main script launches a new python, it is not aware of `ii` and hence `iii`.
`bootstrap_impl=script` does not have this issue as `sys.executable` resolves to a different python (`test.venv/bin/python`) that always is aware of `iii` even when `ii` is out of picture.
(I suspect if we launch with `-S` even `bootstrap_impl=script` will stop working)
**Fix ideas**:
1. use same venv python for `system_python` too
2. use [sitecustomize](https://docs.python.org/3/library/site.html) + PYTHONPATH in bootstrap stage 1 instead of `site.addsitepackage` in bootstrap stage 2. This should propagate.
コントリビューションガイド
調査の方向性
まず python/private/python_bootstrap_template.txt、python/private/stage2_bootstrap_template.py、python/private/site_init_template.py を比較し、system_python がサブプロセス用のインポートパスをどのように構築しているかに注目します。リンクされている最小再現を使って失敗を検証し、その後、bootstrap_impl=system_python の場合に py_binary または py_test のサブプロセスが宣言された依存関係をインポートでき、既存のスクリプトの動作はそのまま維持されることを示す回帰テストを追加します。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- python
- 領域
- build-system
- issue の種類
- バグ
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 活発さ
- 静か
- 明瞭さ
- おおむね明確
- 初心者へのやさしさ
- 48/100