bazel-contrib / bazel-contrib/rules_python
"attempting to create a duplicate library" when multiple python versions with same major+minor are used (different patch version)
- Dominant language
- Starlark
- Stars
- 688
- Forks
- 721
- Avg merge
- 14h 49m
- Merged PRs (30d)
- 81
Description
# 🐞 bug report
### Affected Rule
Maybe `pip.parse`?
### Is this a regression?
It appears to be, yes. I bisected from `1.6.1` to `1.7.0` and found that commit 7b88c87aaab1e4711a8b61d2a47f445052ed6e9a (PR #3243) is the first bad commit.
### Description
We're getting a "duplicate library" error.
```
Error in fail: attempting to create a duplicate library pypi_313_absl_py_py3_none_any_9824a48b for absl_py
```
## 🔬 Minimal Reproduction
**TL;DR: running multiple python versions with the same **minor** version (eg 3.13.4 and 3.13.6) appears to break things.**
I'll see what I can do about making a minimal reproduction.
### Background
We have things set up so that each python target is actually a macro wrapping `py_*` for the current python version and `py_*` for the "pynext" python version. Example:
```starlark
load("@rules_python//python:defs.bzl", _py_binary = "py_binary", _py_test = "py_test")
load("@python_versions//3.13:defs.bzl", _py_binary_next = "py_binary", _py_test_next = "py_test") # py version kept in sync with MODULE.bazel
def pyle_py_binary(**kwargs):
_py_binary(**kwargs)
_py_binary_next(**kwargs)
```
And thus our `MODULE.bazel` file has:
```starlark
PYTHON_VERSION = "3.13.4"
PYNEXT_VERSION = "3.13.6" # must be different from PYTHON_VERSION. Before 1.7.0, a different **patch** version was sufficient
python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
...,
is_default = True,
python_version = PYTHON_VERSION,
)
python.toolchain(
...,
is_default = False,
python_version = PYNEXT_VERSION,
)
use_repo(python, PYTHON_VERSION_NAME, PYNEXT_VERSION_NAME, "python_versions")
pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
experimental_index_url = ...,
experimental_index_url_overrides = ...,
hub_name = "pypi",
python_version = PYTHON_VERSION,
requirements_lock = "//:requirements.txt",
)
pip.parse(
experimental_index_url = ...,
experimental_index_url_overrides = ...,
hub_name = "pypi",
python_version = PYNEXT_VERSION,
requirements_lock = "//:requirements.txt",
)
```
You'll notice that we're using the same requirements.txt lock file and `hub_name` for both current and "pynext" versions.
We then use build/test tags to only build/test the current python version or the "pynext" python version:
```shell
# (default: --test_tag_filters=-pynext, only run against the current python version)
bazel test //...
# optionally test against both versions
bazel test --test_tag_filters= //...
# optionally test against only the "pynext" version
bazel test --test_tag_filters=pynext //...
```
## 🔥 Exception or Error
```
bazel build --nobuild //...
INFO: Invocation ID: 8783fed4-2abe-4061-a080-4b3640de62a1
ERROR: /usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl:185:13: Traceback (most recent call last):
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/extension.bzl", line 390, column 25, in _pip_impl
mods = parse_modules(module_ctx, enable_pipstar = rp_config.enable_pipstar)
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/extension.bzl", line 280, column 30, in parse_modules
builder.pip_parse(
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl", line 58, column 47, in lambda
pip_parse = lambda *a, **k: _pip_parse(self, *a, **k),
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl", line 125, column 22, in _pip_parse
_create_whl_repos(
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl", line 440, column 29, in _create_whl_repos
_add_whl_library(
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl", line 185, column 13, in _add_whl_library
fail("attempting to create a duplicate library {} for {}".format(
Error in fail: attempting to create a duplicate library pypi_313_absl_py_py3_none_any_9824a48b for absl_py
ERROR: error evaluating module extension @@rules_python+//python/extensions:pip.bzl%pip
INFO: Elapsed time: 26.869s
INFO: 0 processes.
ERROR: Build did NOT complete successfully
Loading: 363 packages loaded
currently loading:
Fetching module extension @@rules_python+//python/extensions:pip.bzl%pip; Fetch package lists from PyPI index 26s
```
## 🌍 Your Environment
**Operating System:**
```
gLinux (Debian Testing)
```
**Output of `bazel version`:**
```
$ bazel version
Bazelisk version: v1.26.0
Build label: 8.3.1
Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer
Build time: Mon Jun 30 16:23:40 2025 (1751300620)
Build timestamp: 1751300620
Build timestamp as int: 1751300620
```
**Rules_python version:**
1.7.0
**Anything else relevant?**
Removing any support for "pynext" allows us to bump to 1.7.0. Additionally, bumping `PYNEXT_VERSION` to a different **minor** version than `PYTHON_VERSION` appears to work (eg current=3.13, next=3.14), but I can't fully confirm this yet because our requirements are not compatible with python 3.14.
This might be PEBKAC - if there's a more "correct" way of running multiple python versions please let me know. Our requirements for such are:
+ Do not build/test/run any "pynext" version by default
+ Only need to support versions N and N+1, where N might be a major, minor, or patch version
+ Must use the same requirements.txt lock file.
+ Must use the same pypi hub name
+ Or more accurately: the third-party deps of a target must not change between python versions.
Contributor guide
Assessment
This issue has not been assessed yet.