bazel-contrib / bazel-contrib/rules_python
compile_pip_requirements update and all_whl_requirements seem like a circular dependency
- 主要语言
- Starlark
- 星标
- 688
- 派生
- 721
- 平均合并
- 15 小时 7 分钟
- 30 天内合并 PR
- 76
描述
# 🐞 bug report
### Affected Rule
The issue is caused by the rule:
`compile_pip_requirements`
`all_whl_requirements`
### Is this a regression?
Have not tested previous versions.
### Description
A clear and concise description of the problem...
When trying to switch the index url I noticed that requirements.txt was getting populated with a package that didnt exist in the requirements.txt file when running the `compile_pip_requirements.update` rule. This should have failed from the beginning as the registry for the index_url was bogus. Then with further poking around I ran it again and this time it failed. There are 2 concerns:
1. How is the requirements.txt being filled in with a package when overriding the index url to one that doesnt contain the package?
2. The rule `all_whl_requirements` seems to be loading before the update command causing it to pass with the empty requirements.txt and then fail the second time when running the same command since the package doesnt actually exist in the new index registry.
## 🔬 Minimal Reproduction
`MODULE.bazel`:
```
bazel_dep(name = "rules_python", version = "1.3.0")
python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
configure_coverage_tool = True,
ignore_root_user_error = True,
python_version = "3.11",
)
use_repo(python, "python_3_11")
pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
experimental_index_url = "https://us-python.pkg.dev/null/simple",
hub_name = "pypi",
python_version = "3.11",
requirements_lock = "//:requirements.txt",
)
use_repo(pip, "pypi")
```
`requirements.in`:
```
itsdangerous
```
`requirements.txt` is empty to start.
`BUILD.bazel`:
```
load("@pypi//:requirements.bzl", "all_whl_requirements")
load("@rules_python//python:pip.bzl", "compile_pip_requirements")
compile_pip_requirements(
name = "requirements",
src = "requirements.in",
extra_args = [
"--allow-unsafe",
],
requirements_txt = "requirements.txt",
)
```
To produce error:
1. Execute the command: `bazel run //:requirements.update`
2. Execute the command again: `bazel run //:requirements.update`
## 🔥 Exception or Error
```
$ bazel run //:requirements.update
INFO: Analyzed target //:requirements.update (131 packages loaded, 5471 targets configured).
INFO: Found 1 target...
Target //:requirements.update up-to-date:
bazel-bin/requirements.update
INFO: Elapsed time: 6.987s, Critical Path: 0.59s
INFO: 5 processes: 5 internal.
INFO: Build completed successfully, 5 total actions
INFO: Running command line: bazel-bin/requirements.update '--src=_main/requirements.in' _main/requirements.txt //:requirements.update '--resolver=backtracking' --allow-unsafe --generate-hashes --allow-unsafe
Updating requirements.txt
WARNING: --strip-extras is becoming the default in version 8.0.0. To silence this warning, either use --strip-extras to opt into the new default or use --no-strip-extras to retain the existing behavior.
#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
# bazel run //:requirements.update
#
itsdangerous==2.2.0 \
--hash=sha256:c6242fc49e35958c8b15141343aa660db5fc54d4f13a1db01a3f5891b98700ef \
--hash=sha256:e0050c0b7da1eea53ffaf149c0cfbb5c6e2e2b69c4bef22c81fa6eb73e5f6173
# via -r requirements.in
$ bazel run //:requirements.update
WARNING: Download from https://us-python.pkg.dev/null/simple/itsdangerous/ failed: class java.io.FileNotFoundException GET returned 404 Not Found
ERROR: ~/.cache/bazel/_bazel_user/f7d37797d74a062b8f39bc84d0ebc222/external/rules_python~/python/private/pypi/simpleapi_download.bzl:130:14: Traceback (most recent call last):
File "~/.cache/bazel/_bazel_user/f7d37797d74a062b8f39bc84d0ebc222/external/rules_python~/python/private/pypi/extension.bzl", line 615, column 25, in _pip_impl
mods = parse_modules(module_ctx)
File "~/.cache/bazel/_bazel_user/f7d37797d74a062b8f39bc84d0ebc222/external/rules_python~/python/private/pypi/extension.bzl", line 490, column 36, in parse_modules
out = _create_whl_repos(
File "~/.cache/bazel/_bazel_user/f7d37797d74a062b8f39bc84d0ebc222/external/rules_python~/python/private/pypi/extension.bzl", line 155, column 50, in _create_whl_repos
requirements_by_platform = parse_requirements(
File "~/.cache/bazel/_bazel_user/f7d37797d74a062b8f39bc84d0ebc222/external/rules_python~/python/private/pypi/parse_requirements.bzl", line 180, column 36, in parse_requirements
index_urls = get_index_urls(
File "~/.cache/bazel/_bazel_user/f7d37797d74a062b8f39bc84d0ebc222/external/rules_python~/python/private/pypi/extension.bzl", line 470, column 79, in lambda
get_index_urls = lambda ctx, distributions: simpleapi_download(
File "~/.cache/bazel/_bazel_user/f7d37797d74a062b8f39bc84d0ebc222/external/rules_python~/python/private/pypi/simpleapi_download.bzl", line 130, column 14, in simpleapi_download
_fail("Failed to download metadata for {} for from urls: {}".format(
Error in fail: Failed to download metadata for ["itsdangerous"] for from urls: ["https://us-python.pkg.dev/null/simple"]
ERROR: error evaluating module extension pip in @@rules_python~//python/extensions:pip.bzl
INFO: Elapsed time: 0.521s
INFO: 0 processes.
ERROR: Build did NOT complete successfully
ERROR: Build failed. Not running target
FAILED:
Fetching module extension pip in @@rules_python~//python/extensions:pip.bzl; starting
```
## 🌍 Your Environment
**Operating System:**
Linux
**Output of `bazel version`:**
$ bazel version
Bazelisk version: development
Build label: 7.4.0
Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer
Build time: Tue Oct 22 17:24:25 2024 (1729617865)
Build timestamp: 1729617865
Build timestamp as int: 1729617865
**Rules_python version:**
1.3.0
**Anything else relevant?**
贡献指南
调研方向
使用所示的 MODULE.bazel、BUILD.bazel、requirements.in 和空的 requirements.txt,重现 issue 中的两个 `bazel run //:requirements.update` 命令。然后阅读报告的堆栈跟踪附近的 `python/private/pypi/extension.bzl`、`parse_requirements.bzl` 和 `simpleapi_download.bzl`;完成的标准是以一致的方式处理无效的索引 URL,且不会在第一次运行时生成包或创建循环依赖。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- python
- 领域
- build-system
- Issue 类型
- 缺陷
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 35/100