bazel-contrib / bazel-contrib/rules_python

"attempting to create a duplicate library" when multiple python versions with same major+minor are used (different patch version)

オープン
#3,479 コメント 7 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Starlark
スター
688
フォーク
721
平均マージ
15時間 7分
マージ済み PR(30日)
76

説明

# 🐞 bug report

### Affected Rule

Maybe `pip.parse`?

### Is this a regression?

It appears to be, yes. I bisected from `1.6.1` to `1.7.0` and found that commit 7b88c87aaab1e4711a8b61d2a47f445052ed6e9a (PR #3243) is the first bad commit.

### Description

We're getting a "duplicate library" error.

```
Error in fail: attempting to create a duplicate library pypi_313_absl_py_py3_none_any_9824a48b for absl_py
```

## 🔬 Minimal Reproduction

**TL;DR: running multiple python versions with the same **minor** version (eg 3.13.4 and 3.13.6) appears to break things.**

I'll see what I can do about making a minimal reproduction.

### Background

We have things set up so that each python target is actually a macro wrapping `py_*` for the current python version and `py_*` for the "pynext" python version. Example:

```starlark
load("@rules_python//python:defs.bzl", _py_binary = "py_binary", _py_test = "py_test")
load("@python_versions//3.13:defs.bzl", _py_binary_next = "py_binary", _py_test_next = "py_test") # py version kept in sync with MODULE.bazel
def pyle_py_binary(**kwargs):
_py_binary(**kwargs)
_py_binary_next(**kwargs)
```

And thus our `MODULE.bazel` file has:

```starlark
PYTHON_VERSION = "3.13.4"
PYNEXT_VERSION = "3.13.6" # must be different from PYTHON_VERSION. Before 1.7.0, a different **patch** version was sufficient

python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
...,
is_default = True,
python_version = PYTHON_VERSION,
)
python.toolchain(
...,
is_default = False,
python_version = PYNEXT_VERSION,
)
use_repo(python, PYTHON_VERSION_NAME, PYNEXT_VERSION_NAME, "python_versions")

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
experimental_index_url = ...,
experimental_index_url_overrides = ...,
hub_name = "pypi",
python_version = PYTHON_VERSION,
requirements_lock = "//:requirements.txt",
)
pip.parse(
experimental_index_url = ...,
experimental_index_url_overrides = ...,
hub_name = "pypi",
python_version = PYNEXT_VERSION,
requirements_lock = "//:requirements.txt",
)
```

You'll notice that we're using the same requirements.txt lock file and `hub_name` for both current and "pynext" versions.

We then use build/test tags to only build/test the current python version or the "pynext" python version:

```shell
# (default: --test_tag_filters=-pynext, only run against the current python version)
bazel test //...

# optionally test against both versions
bazel test --test_tag_filters= //...

# optionally test against only the "pynext" version
bazel test --test_tag_filters=pynext //...
```

## 🔥 Exception or Error

```
bazel build --nobuild //...
INFO: Invocation ID: 8783fed4-2abe-4061-a080-4b3640de62a1
ERROR: /usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl:185:13: Traceback (most recent call last):
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/extension.bzl", line 390, column 25, in _pip_impl
mods = parse_modules(module_ctx, enable_pipstar = rp_config.enable_pipstar)
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/extension.bzl", line 280, column 30, in parse_modules
builder.pip_parse(
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl", line 58, column 47, in lambda
pip_parse = lambda *a, **k: _pip_parse(self, *a, **k),
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl", line 125, column 22, in _pip_parse
_create_whl_repos(
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl", line 440, column 29, in _create_whl_repos
_add_whl_library(
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl", line 185, column 13, in _add_whl_library
fail("attempting to create a duplicate library {} for {}".format(
Error in fail: attempting to create a duplicate library pypi_313_absl_py_py3_none_any_9824a48b for absl_py
ERROR: error evaluating module extension @@rules_python+//python/extensions:pip.bzl%pip
INFO: Elapsed time: 26.869s
INFO: 0 processes.
ERROR: Build did NOT complete successfully
Loading: 363 packages loaded
currently loading:
Fetching module extension @@rules_python+//python/extensions:pip.bzl%pip; Fetch package lists from PyPI index 26s
```

## 🌍 Your Environment

**Operating System:**

```
gLinux (Debian Testing)
```

**Output of `bazel version`:**

```
$ bazel version
Bazelisk version: v1.26.0
Build label: 8.3.1
Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer
Build time: Mon Jun 30 16:23:40 2025 (1751300620)
Build timestamp: 1751300620
Build timestamp as int: 1751300620
```

**Rules_python version:**

1.7.0

**Anything else relevant?**

Removing any support for "pynext" allows us to bump to 1.7.0. Additionally, bumping `PYNEXT_VERSION` to a different **minor** version than `PYTHON_VERSION` appears to work (eg current=3.13, next=3.14), but I can't fully confirm this yet because our requirements are not compatible with python 3.14.

This might be PEBKAC - if there's a more "correct" way of running multiple python versions please let me know. Our requirements for such are:

+ Do not build/test/run any "pynext" version by default
+ Only need to support versions N and N+1, where N might be a major, minor, or patch version
+ Must use the same requirements.txt lock file.
+ Must use the same pypi hub name
+ Or more accurately: the third-party deps of a target must not change between python versions.

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

Start with python/private/pypi/hub_builder.bzl, especially the duplicate-library failure at line 185, and trace how python/private/pypi/extension.bzl processes multiple pip.parse calls. Reproduce the Bazel build using two Python patch versions, the same requirements lock file, and the same hub name. Done means those configurations no longer fail while supporting the reported N and N+1 version workflow.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
python
領域
build-system
issue の種類
バグ
難易度
4/5
見積もり時間
3〜5日
活発さ
停滞
明瞭さ
おおむね明確
初心者へのやさしさ
42/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。