bazel-contrib / bazel-contrib/rules_python
compile_pip_requirements + 3.11 failing on some packages
- 主要言語
- Starlark
- スター
- 688
- フォーク
- 721
- 平均マージ
- 15時間 7分
- マージ済み PR(30日)
- 76
説明
# 🐞 bug report
### Affected Rule
The issue is caused by the rule: `compile_pip_requirements`
### Is this a regression?
I don't think this is a regression in rules_python, but this works with Python 3.10 and does not work with 3.11.
### Description
When I try to use `compile_pip_requirements` to recompile a lockfile with certain packages it cannot handle it. I would have thought the libraries were wrong, however a) `pip install` works in a non-Bazel venv and b) if I manually resolve the lockfile then `pip_parse` successfully can parse it and the dependencies work in my `py_binary` targets.
This seems similar-but-different to https://github.com/bazelbuild/rules_python/issues/914, but `enable_implicit_namespace_pkgs` does not seem to be an argument to `compile_pip_requirements` and anyway `pip_parse` has no problem parsing this package. Plus adding or removing `--incompatible_default_to_explicit_init_py` has no effect on this problem.
## 🔬 Minimal Reproduction
`WORKSPACE`:
```
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "rules_python",
sha256 = "a644da969b6824cc87f8fe7b18101a8a6c57da5db39caa6566ec6109f37d2141",
strip_prefix = "rules_python-0.20.0",
url = "https://github.com/bazelbuild/rules_python/releases/download/0.20.0/rules_python-0.20.0.tar.gz",
)
load("@rules_python//python:repositories.bzl", "py_repositories")
py_repositories()
load("@rules_python//python:repositories.bzl", "python_register_toolchains")
python_register_toolchains(
name = "python3_11",
python_version = "3.11",
)
load("@python3_11//:defs.bzl", "interpreter")
load("@rules_python//python:pip.bzl", "pip_parse")
pip_parse(
name = "python_deps",
python_interpreter_target = interpreter,
requirements_lock = "//:requirements_actual.txt",
)
load("@python_deps//:requirements.bzl", install_deps = "install_deps")
install_deps()
```
`BUILD`:
```
load("@rules_python//python:pip.bzl", "compile_pip_requirements")
package(default_visibility = ["//visibility:public"])
compile_pip_requirements(
name = "requirements_lock",
extra_args = ["--allow-unsafe"],
)
```
`requirements_lock.in`:
```
docopt==0.6.2
```
Then run `bazel run //:requirements_lock.update`
If you create `requirements_actual.txt` with the same content as `requirements_lock.in`, then you can `bazel build @python_deps_docopt//:pkg` or use it as a dependency of a `py_binary` target succesfully.
## 🔥 Exception or Error
Updating ./requirements_lock.txt
WARNING: the legacy dependency resolver is deprecated and will be removed in future versions of pip-tools. The default resolver will be changed to 'backtracking' in pip-tools 7.0.0. Specify --resolver=backtracking to silence this warning.
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [6 lines of output]
Traceback (most recent call last):
File "", line 2, in
File "", line 34, in
File "/tmp/pip-resolver-fahvht28/docopt/setup.py", line 3, in
from docopt import __version__
ModuleNotFoundError: No module named 'docopt'
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
Traceback (most recent call last):
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/operations/build/metadata_legacy.py", line 64, in generate_metadata
call_subprocess(
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/utils/subprocess.py", line 224, in call_subprocess
raise error
pip._internal.exceptions.InstallationSubprocessError: python setup.py egg_info exited with 1
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/rules_python/python/pip_install/tools/dependency_resolver/dependency_resolver.py", line 143, in
cli()
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__click/click/core.py", line 1137, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__click/click/core.py", line 1062, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__click/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__click/click/core.py", line 763, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__click/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip_tools/piptools/scripts/compile.py", line 580, in cli
results = resolver.resolve(max_rounds=max_rounds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip_tools/piptools/resolver.py", line 253, in resolve
has_changed, best_matches = self._resolve_one_round()
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip_tools/piptools/resolver.py", line 343, in _resolve_one_round
their_constraints.extend(self._iter_dependencies(best_match))
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip_tools/piptools/resolver.py", line 456, in _iter_dependencies
dependencies = self.repository.get_dependencies(ireq)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip_tools/piptools/repositories/local.py", line 87, in get_dependencies
return self.repository.get_dependencies(ireq)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip_tools/piptools/repositories/pypi.py", line 238, in get_dependencies
self._dependencies_cache[ireq] = self.resolve_reqs(
^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip_tools/piptools/repositories/pypi.py", line 201, in resolve_reqs
results = resolver._resolve_one(reqset, ireq)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/resolution/legacy/resolver.py", line 509, in _resolve_one
dist = self._get_dist_for(req_to_install)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/resolution/legacy/resolver.py", line 462, in _get_dist_for
dist = self.preparer.prepare_linked_requirement(req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/operations/prepare.py", line 491, in prepare_linked_requirement
return self._prepare_linked_requirement(req, parallel_builds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/operations/prepare.py", line 577, in _prepare_linked_requirement
dist = _get_prepared_distribution(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/operations/prepare.py", line 69, in _get_prepared_distribution
abstract_dist.prepare_distribution_metadata(
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/distributions/sdist.py", line 61, in prepare_distribution_metadata
self.req.prepare_metadata()
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/req/req_install.py", line 541, in prepare_metadata
self.metadata_directory = generate_metadata_legacy(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/operations/build/metadata_legacy.py", line 71, in generate_metadata
raise MetadataGenerationFailed(package_details=details) from error
pip._internal.exceptions.MetadataGenerationFailed: metadata generation failed
## 🌍 Your Environment
**Operating System:**
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.2 LTS"
**Output of `bazel version`:**
Bazelisk version: v1.14.0
Build label: 6.1.1
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Wed Mar 15 15:44:56 2023 (1678895096)
Build timestamp: 1678895096
Build timestamp as int: 1678895096
**Rules_python version:**
0.20.0
**Anything else relevant?**
I'm also seeing this with another package, `umi-tools`, though there are some other 3.10->3.11 problems there I think so I felt `docopt` was a cleaner demonstration.
PS thank you so much for maintaining this project! Super amazing to have so much complexity hidden behind relatively simple rules.
コントリビューションガイド
調査の方向性
rules_python 0.20.0、Python 3.11、および docopt 要件を使用して WORKSPACE と BUILD の例を再現し、その後 `bazel run //:requirements_lock.update` を実行します。`compile_pip_requirements` エントリポイントと traceback の `dependency_resolver.py` を調査します。コマンドが成功し、lockfile が生成され、結果として得られた依存関係を Bazel ターゲットで引き続き使用できれば完了です。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- python
- 領域
- build-system
- issue の種類
- バグ
- 難易度
- 3/5
- 見積もり時間
- 1〜2日
- 活発さ
- 活発
- 明瞭さ
- おおむね明確
- 初心者へのやさしさ
- 64/100