bazel-contrib / bazel-contrib/rules_python

"attempting to create a duplicate library" when multiple python versions with same major+minor are used (different patch version)

未关闭
#3,479 7 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Starlark
星标
688
派生
721
平均合并
15 小时 7 分钟
30 天内合并 PR
76

描述

# 🐞 bug report

### Affected Rule

Maybe `pip.parse`?

### Is this a regression?

It appears to be, yes. I bisected from `1.6.1` to `1.7.0` and found that commit 7b88c87aaab1e4711a8b61d2a47f445052ed6e9a (PR #3243) is the first bad commit.

### Description

We're getting a "duplicate library" error.

```
Error in fail: attempting to create a duplicate library pypi_313_absl_py_py3_none_any_9824a48b for absl_py
```

## 🔬 Minimal Reproduction

**TL;DR: running multiple python versions with the same **minor** version (eg 3.13.4 and 3.13.6) appears to break things.**

I'll see what I can do about making a minimal reproduction.

### Background

We have things set up so that each python target is actually a macro wrapping `py_*` for the current python version and `py_*` for the "pynext" python version. Example:

```starlark
load("@rules_python//python:defs.bzl", _py_binary = "py_binary", _py_test = "py_test")
load("@python_versions//3.13:defs.bzl", _py_binary_next = "py_binary", _py_test_next = "py_test") # py version kept in sync with MODULE.bazel
def pyle_py_binary(**kwargs):
_py_binary(**kwargs)
_py_binary_next(**kwargs)
```

And thus our `MODULE.bazel` file has:

```starlark
PYTHON_VERSION = "3.13.4"
PYNEXT_VERSION = "3.13.6" # must be different from PYTHON_VERSION. Before 1.7.0, a different **patch** version was sufficient

python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
...,
is_default = True,
python_version = PYTHON_VERSION,
)
python.toolchain(
...,
is_default = False,
python_version = PYNEXT_VERSION,
)
use_repo(python, PYTHON_VERSION_NAME, PYNEXT_VERSION_NAME, "python_versions")

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
experimental_index_url = ...,
experimental_index_url_overrides = ...,
hub_name = "pypi",
python_version = PYTHON_VERSION,
requirements_lock = "//:requirements.txt",
)
pip.parse(
experimental_index_url = ...,
experimental_index_url_overrides = ...,
hub_name = "pypi",
python_version = PYNEXT_VERSION,
requirements_lock = "//:requirements.txt",
)
```

You'll notice that we're using the same requirements.txt lock file and `hub_name` for both current and "pynext" versions.

We then use build/test tags to only build/test the current python version or the "pynext" python version:

```shell
# (default: --test_tag_filters=-pynext, only run against the current python version)
bazel test //...

# optionally test against both versions
bazel test --test_tag_filters= //...

# optionally test against only the "pynext" version
bazel test --test_tag_filters=pynext //...
```

## 🔥 Exception or Error

```
bazel build --nobuild //...
INFO: Invocation ID: 8783fed4-2abe-4061-a080-4b3640de62a1
ERROR: /usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl:185:13: Traceback (most recent call last):
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/extension.bzl", line 390, column 25, in _pip_impl
mods = parse_modules(module_ctx, enable_pipstar = rp_config.enable_pipstar)
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/extension.bzl", line 280, column 30, in parse_modules
builder.pip_parse(
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl", line 58, column 47, in lambda
pip_parse = lambda *a, **k: _pip_parse(self, *a, **k),
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl", line 125, column 22, in _pip_parse
_create_whl_repos(
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl", line 440, column 29, in _create_whl_repos
_add_whl_library(
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl", line 185, column 13, in _add_whl_library
fail("attempting to create a duplicate library {} for {}".format(
Error in fail: attempting to create a duplicate library pypi_313_absl_py_py3_none_any_9824a48b for absl_py
ERROR: error evaluating module extension @@rules_python+//python/extensions:pip.bzl%pip
INFO: Elapsed time: 26.869s
INFO: 0 processes.
ERROR: Build did NOT complete successfully
Loading: 363 packages loaded
currently loading:
Fetching module extension @@rules_python+//python/extensions:pip.bzl%pip; Fetch package lists from PyPI index 26s
```

## 🌍 Your Environment

**Operating System:**

```
gLinux (Debian Testing)
```

**Output of `bazel version`:**

```
$ bazel version
Bazelisk version: v1.26.0
Build label: 8.3.1
Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer
Build time: Mon Jun 30 16:23:40 2025 (1751300620)
Build timestamp: 1751300620
Build timestamp as int: 1751300620
```

**Rules_python version:**

1.7.0

**Anything else relevant?**

Removing any support for "pynext" allows us to bump to 1.7.0. Additionally, bumping `PYNEXT_VERSION` to a different **minor** version than `PYTHON_VERSION` appears to work (eg current=3.13, next=3.14), but I can't fully confirm this yet because our requirements are not compatible with python 3.14.

This might be PEBKAC - if there's a more "correct" way of running multiple python versions please let me know. Our requirements for such are:

+ Do not build/test/run any "pynext" version by default
+ Only need to support versions N and N+1, where N might be a major, minor, or patch version
+ Must use the same requirements.txt lock file.
+ Must use the same pypi hub name
+ Or more accurately: the third-party deps of a target must not change between python versions.

贡献指南

打开贡献指南

调研方向

从 python/private/pypi/hub_builder.bzl 开始,重点关注第 185 行的重复库失败,并跟踪 python/private/pypi/extension.bzl 如何处理多个 pip.parse 调用。使用两个 Python 补丁版本、同一个 requirements 锁定文件和同一个 hub 名称,重现 Bazel 构建。当这些配置不再失败,同时支持所报告的 N 和 N+1 版本工作流时,即表示完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
build-system
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
42/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。