bazel-contrib / bazel-contrib/rules_python

compile_pip_requirements + 3.11 failing on some packages

Đang mở
#1,140 8 bình luận 3 reaction 0 người được giao Xem trên GitHub
Can Close?
Ngôn ngữ chính
Starlark
Star
688
Fork
721
Merge trung bình
15 giờ 7 phút
Pull request đã merge (30 ngày)
76

Mô tả

# 🐞 bug report

### Affected Rule

The issue is caused by the rule: `compile_pip_requirements`

### Is this a regression?

I don't think this is a regression in rules_python, but this works with Python 3.10 and does not work with 3.11.

### Description

When I try to use `compile_pip_requirements` to recompile a lockfile with certain packages it cannot handle it. I would have thought the libraries were wrong, however a) `pip install` works in a non-Bazel venv and b) if I manually resolve the lockfile then `pip_parse` successfully can parse it and the dependencies work in my `py_binary` targets.

This seems similar-but-different to https://github.com/bazelbuild/rules_python/issues/914, but `enable_implicit_namespace_pkgs` does not seem to be an argument to `compile_pip_requirements` and anyway `pip_parse` has no problem parsing this package. Plus adding or removing `--incompatible_default_to_explicit_init_py` has no effect on this problem.

## 🔬 Minimal Reproduction

`WORKSPACE`:

```
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
name = "rules_python",
sha256 = "a644da969b6824cc87f8fe7b18101a8a6c57da5db39caa6566ec6109f37d2141",
strip_prefix = "rules_python-0.20.0",
url = "https://github.com/bazelbuild/rules_python/releases/download/0.20.0/rules_python-0.20.0.tar.gz",
)

load("@rules_python//python:repositories.bzl", "py_repositories")

py_repositories()

load("@rules_python//python:repositories.bzl", "python_register_toolchains")

python_register_toolchains(
name = "python3_11",
python_version = "3.11",
)

load("@python3_11//:defs.bzl", "interpreter")
load("@rules_python//python:pip.bzl", "pip_parse")

pip_parse(
name = "python_deps",
python_interpreter_target = interpreter,
requirements_lock = "//:requirements_actual.txt",
)

load("@python_deps//:requirements.bzl", install_deps = "install_deps")

install_deps()
```

`BUILD`:

```
load("@rules_python//python:pip.bzl", "compile_pip_requirements")

package(default_visibility = ["//visibility:public"])

compile_pip_requirements(
name = "requirements_lock",
extra_args = ["--allow-unsafe"],
)
```

`requirements_lock.in`:

```
docopt==0.6.2
```

Then run `bazel run //:requirements_lock.update`

If you create `requirements_actual.txt` with the same content as `requirements_lock.in`, then you can `bazel build @python_deps_docopt//:pkg` or use it as a dependency of a `py_binary` target succesfully.

## 🔥 Exception or Error



Updating ./requirements_lock.txt
WARNING: the legacy dependency resolver is deprecated and will be removed in future versions of pip-tools. The default resolver will be changed to 'backtracking' in pip-tools 7.0.0. Specify --resolver=backtracking to silence this warning.
error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [6 lines of output]
Traceback (most recent call last):
File "", line 2, in
File "", line 34, in
File "/tmp/pip-resolver-fahvht28/docopt/setup.py", line 3, in
from docopt import __version__
ModuleNotFoundError: No module named 'docopt'
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
Traceback (most recent call last):
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/operations/build/metadata_legacy.py", line 64, in generate_metadata
call_subprocess(
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/utils/subprocess.py", line 224, in call_subprocess
raise error
pip._internal.exceptions.InstallationSubprocessError: python setup.py egg_info exited with 1

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/rules_python/python/pip_install/tools/dependency_resolver/dependency_resolver.py", line 143, in
cli()
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__click/click/core.py", line 1137, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__click/click/core.py", line 1062, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__click/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__click/click/core.py", line 763, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__click/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip_tools/piptools/scripts/compile.py", line 580, in cli
results = resolver.resolve(max_rounds=max_rounds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip_tools/piptools/resolver.py", line 253, in resolve
has_changed, best_matches = self._resolve_one_round()
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip_tools/piptools/resolver.py", line 343, in _resolve_one_round
their_constraints.extend(self._iter_dependencies(best_match))
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip_tools/piptools/resolver.py", line 456, in _iter_dependencies
dependencies = self.repository.get_dependencies(ireq)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip_tools/piptools/repositories/local.py", line 87, in get_dependencies
return self.repository.get_dependencies(ireq)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip_tools/piptools/repositories/pypi.py", line 238, in get_dependencies
self._dependencies_cache[ireq] = self.resolve_reqs(
^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip_tools/piptools/repositories/pypi.py", line 201, in resolve_reqs
results = resolver._resolve_one(reqset, ireq)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/resolution/legacy/resolver.py", line 509, in _resolve_one
dist = self._get_dist_for(req_to_install)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/resolution/legacy/resolver.py", line 462, in _get_dist_for
dist = self.preparer.prepare_linked_requirement(req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/operations/prepare.py", line 491, in prepare_linked_requirement
return self._prepare_linked_requirement(req, parallel_builds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/operations/prepare.py", line 577, in _prepare_linked_requirement
dist = _get_prepared_distribution(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/operations/prepare.py", line 69, in _get_prepared_distribution
abstract_dist.prepare_distribution_metadata(
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/distributions/sdist.py", line 61, in prepare_distribution_metadata
self.req.prepare_metadata()
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/req/req_install.py", line 541, in prepare_metadata
self.metadata_directory = generate_metadata_legacy(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/operations/build/metadata_legacy.py", line 71, in generate_metadata
raise MetadataGenerationFailed(package_details=details) from error
pip._internal.exceptions.MetadataGenerationFailed: metadata generation failed

## 🌍 Your Environment

**Operating System:**

  

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.2 LTS"

**Output of `bazel version`:**

  

Bazelisk version: v1.14.0
Build label: 6.1.1
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Wed Mar 15 15:44:56 2023 (1678895096)
Build timestamp: 1678895096
Build timestamp as int: 1678895096

**Rules_python version:**

  

0.20.0

**Anything else relevant?**

I'm also seeing this with another package, `umi-tools`, though there are some other 3.10->3.11 problems there I think so I felt `docopt` was a cleaner demonstration.

PS thank you so much for maintaining this project! Super amazing to have so much complexity hidden behind relatively simple rules.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Tái hiện ví dụ WORKSPACE và BUILD với rules_python 0.20.0, Python 3.11 và yêu cầu docopt, sau đó chạy `bazel run //:requirements_lock.update`. Kiểm tra entry point `compile_pip_requirements` và `dependency_resolver.py` trong traceback; hoàn tất khi lệnh chạy thành công, tạo lockfile và dependency kết quả vẫn có thể được các Bazel target sử dụng.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
python
Lĩnh vực
build-system
Loại issue
Lỗi
Độ khó
3/5
Thời gian dự kiến
1-2 ngày
Mức độ hoạt động
Sôi nổi
Độ rõ ràng
Khá rõ ràng
Mức phù hợp với người mới
64/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.