bazel-contrib / bazel-contrib/rules_python

compile_pip_requirements + 3.11 failing on some packages

Open
#1,140 8 comments 3 reactions 0 assignees View on GitHub
Can Close?
Dominant language
Starlark
Stars
688
Forks
721
Avg merge
15h 7m
Merged PRs (30d)
76

Description

# 🐞 bug report

### Affected Rule

The issue is caused by the rule: `compile_pip_requirements`

### Is this a regression?

I don't think this is a regression in rules_python, but this works with Python 3.10 and does not work with 3.11.

### Description

When I try to use `compile_pip_requirements` to recompile a lockfile with certain packages it cannot handle it. I would have thought the libraries were wrong, however a) `pip install` works in a non-Bazel venv and b) if I manually resolve the lockfile then `pip_parse` successfully can parse it and the dependencies work in my `py_binary` targets.

This seems similar-but-different to https://github.com/bazelbuild/rules_python/issues/914, but `enable_implicit_namespace_pkgs` does not seem to be an argument to `compile_pip_requirements` and anyway `pip_parse` has no problem parsing this package. Plus adding or removing `--incompatible_default_to_explicit_init_py` has no effect on this problem.

## 🔬 Minimal Reproduction

`WORKSPACE`:

```
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
name = "rules_python",
sha256 = "a644da969b6824cc87f8fe7b18101a8a6c57da5db39caa6566ec6109f37d2141",
strip_prefix = "rules_python-0.20.0",
url = "https://github.com/bazelbuild/rules_python/releases/download/0.20.0/rules_python-0.20.0.tar.gz",
)

load("@rules_python//python:repositories.bzl", "py_repositories")

py_repositories()

load("@rules_python//python:repositories.bzl", "python_register_toolchains")

python_register_toolchains(
name = "python3_11",
python_version = "3.11",
)

load("@python3_11//:defs.bzl", "interpreter")
load("@rules_python//python:pip.bzl", "pip_parse")

pip_parse(
name = "python_deps",
python_interpreter_target = interpreter,
requirements_lock = "//:requirements_actual.txt",
)

load("@python_deps//:requirements.bzl", install_deps = "install_deps")

install_deps()
```

`BUILD`:

```
load("@rules_python//python:pip.bzl", "compile_pip_requirements")

package(default_visibility = ["//visibility:public"])

compile_pip_requirements(
name = "requirements_lock",
extra_args = ["--allow-unsafe"],
)
```

`requirements_lock.in`:

```
docopt==0.6.2
```

Then run `bazel run //:requirements_lock.update`

If you create `requirements_actual.txt` with the same content as `requirements_lock.in`, then you can `bazel build @python_deps_docopt//:pkg` or use it as a dependency of a `py_binary` target succesfully.

## 🔥 Exception or Error



Updating ./requirements_lock.txt
WARNING: the legacy dependency resolver is deprecated and will be removed in future versions of pip-tools. The default resolver will be changed to 'backtracking' in pip-tools 7.0.0. Specify --resolver=backtracking to silence this warning.
error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [6 lines of output]
Traceback (most recent call last):
File "", line 2, in
File "", line 34, in
File "/tmp/pip-resolver-fahvht28/docopt/setup.py", line 3, in
from docopt import __version__
ModuleNotFoundError: No module named 'docopt'
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
Traceback (most recent call last):
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/operations/build/metadata_legacy.py", line 64, in generate_metadata
call_subprocess(
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/utils/subprocess.py", line 224, in call_subprocess
raise error
pip._internal.exceptions.InstallationSubprocessError: python setup.py egg_info exited with 1

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/rules_python/python/pip_install/tools/dependency_resolver/dependency_resolver.py", line 143, in
cli()
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__click/click/core.py", line 1137, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__click/click/core.py", line 1062, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__click/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__click/click/core.py", line 763, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__click/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip_tools/piptools/scripts/compile.py", line 580, in cli
results = resolver.resolve(max_rounds=max_rounds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip_tools/piptools/resolver.py", line 253, in resolve
has_changed, best_matches = self._resolve_one_round()
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip_tools/piptools/resolver.py", line 343, in _resolve_one_round
their_constraints.extend(self._iter_dependencies(best_match))
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip_tools/piptools/resolver.py", line 456, in _iter_dependencies
dependencies = self.repository.get_dependencies(ireq)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip_tools/piptools/repositories/local.py", line 87, in get_dependencies
return self.repository.get_dependencies(ireq)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip_tools/piptools/repositories/pypi.py", line 238, in get_dependencies
self._dependencies_cache[ireq] = self.resolve_reqs(
^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip_tools/piptools/repositories/pypi.py", line 201, in resolve_reqs
results = resolver._resolve_one(reqset, ireq)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/resolution/legacy/resolver.py", line 509, in _resolve_one
dist = self._get_dist_for(req_to_install)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/resolution/legacy/resolver.py", line 462, in _get_dist_for
dist = self.preparer.prepare_linked_requirement(req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/operations/prepare.py", line 491, in prepare_linked_requirement
return self._prepare_linked_requirement(req, parallel_builds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/operations/prepare.py", line 577, in _prepare_linked_requirement
dist = _get_prepared_distribution(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/operations/prepare.py", line 69, in _get_prepared_distribution
abstract_dist.prepare_distribution_metadata(
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/distributions/sdist.py", line 61, in prepare_distribution_metadata
self.req.prepare_metadata()
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/req/req_install.py", line 541, in prepare_metadata
self.metadata_directory = generate_metadata_legacy(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/operations/build/metadata_legacy.py", line 71, in generate_metadata
raise MetadataGenerationFailed(package_details=details) from error
pip._internal.exceptions.MetadataGenerationFailed: metadata generation failed

## 🌍 Your Environment

**Operating System:**

  

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.2 LTS"

**Output of `bazel version`:**

  

Bazelisk version: v1.14.0
Build label: 6.1.1
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Wed Mar 15 15:44:56 2023 (1678895096)
Build timestamp: 1678895096
Build timestamp as int: 1678895096

**Rules_python version:**

  

0.20.0

**Anything else relevant?**

I'm also seeing this with another package, `umi-tools`, though there are some other 3.10->3.11 problems there I think so I felt `docopt` was a cleaner demonstration.

PS thank you so much for maintaining this project! Super amazing to have so much complexity hidden behind relatively simple rules.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.