bazel-contrib / bazel-contrib/rules_python
"attempting to create a duplicate library" when multiple python versions with same major+minor are used (different patch version)
- Ngôn ngữ chính
- Starlark
- Star
- 688
- Fork
- 721
- Merge trung bình
- 15 giờ 7 phút
- Pull request đã merge (30 ngày)
- 76
Mô tả
# 🐞 bug report
### Affected Rule
Maybe `pip.parse`?
### Is this a regression?
It appears to be, yes. I bisected from `1.6.1` to `1.7.0` and found that commit 7b88c87aaab1e4711a8b61d2a47f445052ed6e9a (PR #3243) is the first bad commit.
### Description
We're getting a "duplicate library" error.
```
Error in fail: attempting to create a duplicate library pypi_313_absl_py_py3_none_any_9824a48b for absl_py
```
## 🔬 Minimal Reproduction
**TL;DR: running multiple python versions with the same **minor** version (eg 3.13.4 and 3.13.6) appears to break things.**
I'll see what I can do about making a minimal reproduction.
### Background
We have things set up so that each python target is actually a macro wrapping `py_*` for the current python version and `py_*` for the "pynext" python version. Example:
```starlark
load("@rules_python//python:defs.bzl", _py_binary = "py_binary", _py_test = "py_test")
load("@python_versions//3.13:defs.bzl", _py_binary_next = "py_binary", _py_test_next = "py_test") # py version kept in sync with MODULE.bazel
def pyle_py_binary(**kwargs):
_py_binary(**kwargs)
_py_binary_next(**kwargs)
```
And thus our `MODULE.bazel` file has:
```starlark
PYTHON_VERSION = "3.13.4"
PYNEXT_VERSION = "3.13.6" # must be different from PYTHON_VERSION. Before 1.7.0, a different **patch** version was sufficient
python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
...,
is_default = True,
python_version = PYTHON_VERSION,
)
python.toolchain(
...,
is_default = False,
python_version = PYNEXT_VERSION,
)
use_repo(python, PYTHON_VERSION_NAME, PYNEXT_VERSION_NAME, "python_versions")
pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
experimental_index_url = ...,
experimental_index_url_overrides = ...,
hub_name = "pypi",
python_version = PYTHON_VERSION,
requirements_lock = "//:requirements.txt",
)
pip.parse(
experimental_index_url = ...,
experimental_index_url_overrides = ...,
hub_name = "pypi",
python_version = PYNEXT_VERSION,
requirements_lock = "//:requirements.txt",
)
```
You'll notice that we're using the same requirements.txt lock file and `hub_name` for both current and "pynext" versions.
We then use build/test tags to only build/test the current python version or the "pynext" python version:
```shell
# (default: --test_tag_filters=-pynext, only run against the current python version)
bazel test //...
# optionally test against both versions
bazel test --test_tag_filters= //...
# optionally test against only the "pynext" version
bazel test --test_tag_filters=pynext //...
```
## 🔥 Exception or Error
```
bazel build --nobuild //...
INFO: Invocation ID: 8783fed4-2abe-4061-a080-4b3640de62a1
ERROR: /usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl:185:13: Traceback (most recent call last):
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/extension.bzl", line 390, column 25, in _pip_impl
mods = parse_modules(module_ctx, enable_pipstar = rp_config.enable_pipstar)
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/extension.bzl", line 280, column 30, in parse_modules
builder.pip_parse(
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl", line 58, column 47, in lambda
pip_parse = lambda *a, **k: _pip_parse(self, *a, **k),
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl", line 125, column 22, in _pip_parse
_create_whl_repos(
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl", line 440, column 29, in _create_whl_repos
_add_whl_library(
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl", line 185, column 13, in _add_whl_library
fail("attempting to create a duplicate library {} for {}".format(
Error in fail: attempting to create a duplicate library pypi_313_absl_py_py3_none_any_9824a48b for absl_py
ERROR: error evaluating module extension @@rules_python+//python/extensions:pip.bzl%pip
INFO: Elapsed time: 26.869s
INFO: 0 processes.
ERROR: Build did NOT complete successfully
Loading: 363 packages loaded
currently loading:
Fetching module extension @@rules_python+//python/extensions:pip.bzl%pip; Fetch package lists from PyPI index 26s
```
## 🌍 Your Environment
**Operating System:**
```
gLinux (Debian Testing)
```
**Output of `bazel version`:**
```
$ bazel version
Bazelisk version: v1.26.0
Build label: 8.3.1
Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer
Build time: Mon Jun 30 16:23:40 2025 (1751300620)
Build timestamp: 1751300620
Build timestamp as int: 1751300620
```
**Rules_python version:**
1.7.0
**Anything else relevant?**
Removing any support for "pynext" allows us to bump to 1.7.0. Additionally, bumping `PYNEXT_VERSION` to a different **minor** version than `PYTHON_VERSION` appears to work (eg current=3.13, next=3.14), but I can't fully confirm this yet because our requirements are not compatible with python 3.14.
This might be PEBKAC - if there's a more "correct" way of running multiple python versions please let me know. Our requirements for such are:
+ Do not build/test/run any "pynext" version by default
+ Only need to support versions N and N+1, where N might be a major, minor, or patch version
+ Must use the same requirements.txt lock file.
+ Must use the same pypi hub name
+ Or more accurately: the third-party deps of a target must not change between python versions.
Hướng dẫn đóng góp
Hướng nghiên cứu
Bắt đầu với python/private/pypi/hub_builder.bzl, đặc biệt là lỗi do thư viện trùng lặp ở dòng 185, và theo dõi cách python/private/pypi/extension.bzl xử lý nhiều lệnh gọi pip.parse. Tái hiện bản build Bazel bằng cách sử dụng hai phiên bản bản vá của Python, cùng một tệp khóa requirements và cùng một tên hub. Được xem là hoàn tất khi các cấu hình đó không còn thất bại và vẫn hỗ trợ workflow phiên bản N và N+1 đã được báo cáo.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- python
- Lĩnh vực
- build-system
- Loại issue
- Lỗi
- Độ khó
- 4/5
- Thời gian dự kiến
- 3-5 ngày
- Mức độ hoạt động
- Đình trệ
- Độ rõ ràng
- Khá rõ ràng
- Mức phù hợp với người mới
- 42/100