bazel-contrib / bazel-contrib/rules_python

"attempting to create a duplicate library" when multiple python versions with same major+minor are used (different patch version)

Open
#3,479 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Starlark
Stars
688
Forks
721
Avg merge
14h 49m
Merged PRs (30d)
81

Description

# 🐞 bug report

### Affected Rule

Maybe `pip.parse`?

### Is this a regression?

It appears to be, yes. I bisected from `1.6.1` to `1.7.0` and found that commit 7b88c87aaab1e4711a8b61d2a47f445052ed6e9a (PR #3243) is the first bad commit.

### Description

We're getting a "duplicate library" error.

```
Error in fail: attempting to create a duplicate library pypi_313_absl_py_py3_none_any_9824a48b for absl_py
```

## 🔬 Minimal Reproduction

**TL;DR: running multiple python versions with the same **minor** version (eg 3.13.4 and 3.13.6) appears to break things.**

I'll see what I can do about making a minimal reproduction.

### Background

We have things set up so that each python target is actually a macro wrapping `py_*` for the current python version and `py_*` for the "pynext" python version. Example:

```starlark
load("@rules_python//python:defs.bzl", _py_binary = "py_binary", _py_test = "py_test")
load("@python_versions//3.13:defs.bzl", _py_binary_next = "py_binary", _py_test_next = "py_test") # py version kept in sync with MODULE.bazel
def pyle_py_binary(**kwargs):
_py_binary(**kwargs)
_py_binary_next(**kwargs)
```

And thus our `MODULE.bazel` file has:

```starlark
PYTHON_VERSION = "3.13.4"
PYNEXT_VERSION = "3.13.6" # must be different from PYTHON_VERSION. Before 1.7.0, a different **patch** version was sufficient

python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
...,
is_default = True,
python_version = PYTHON_VERSION,
)
python.toolchain(
...,
is_default = False,
python_version = PYNEXT_VERSION,
)
use_repo(python, PYTHON_VERSION_NAME, PYNEXT_VERSION_NAME, "python_versions")

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
experimental_index_url = ...,
experimental_index_url_overrides = ...,
hub_name = "pypi",
python_version = PYTHON_VERSION,
requirements_lock = "//:requirements.txt",
)
pip.parse(
experimental_index_url = ...,
experimental_index_url_overrides = ...,
hub_name = "pypi",
python_version = PYNEXT_VERSION,
requirements_lock = "//:requirements.txt",
)
```

You'll notice that we're using the same requirements.txt lock file and `hub_name` for both current and "pynext" versions.

We then use build/test tags to only build/test the current python version or the "pynext" python version:

```shell
# (default: --test_tag_filters=-pynext, only run against the current python version)
bazel test //...

# optionally test against both versions
bazel test --test_tag_filters= //...

# optionally test against only the "pynext" version
bazel test --test_tag_filters=pynext //...
```

## 🔥 Exception or Error

```
bazel build --nobuild //...
INFO: Invocation ID: 8783fed4-2abe-4061-a080-4b3640de62a1
ERROR: /usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl:185:13: Traceback (most recent call last):
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/extension.bzl", line 390, column 25, in _pip_impl
mods = parse_modules(module_ctx, enable_pipstar = rp_config.enable_pipstar)
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/extension.bzl", line 280, column 30, in parse_modules
builder.pip_parse(
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl", line 58, column 47, in lambda
pip_parse = lambda *a, **k: _pip_parse(self, *a, **k),
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl", line 125, column 22, in _pip_parse
_create_whl_repos(
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl", line 440, column 29, in _create_whl_repos
_add_whl_library(
File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/0f8c52850e7230283fc2f8033149fba2/external/rules_python+/python/private/pypi/hub_builder.bzl", line 185, column 13, in _add_whl_library
fail("attempting to create a duplicate library {} for {}".format(
Error in fail: attempting to create a duplicate library pypi_313_absl_py_py3_none_any_9824a48b for absl_py
ERROR: error evaluating module extension @@rules_python+//python/extensions:pip.bzl%pip
INFO: Elapsed time: 26.869s
INFO: 0 processes.
ERROR: Build did NOT complete successfully
Loading: 363 packages loaded
currently loading:
Fetching module extension @@rules_python+//python/extensions:pip.bzl%pip; Fetch package lists from PyPI index 26s
```

## 🌍 Your Environment

**Operating System:**

```
gLinux (Debian Testing)
```

**Output of `bazel version`:**

```
$ bazel version
Bazelisk version: v1.26.0
Build label: 8.3.1
Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer
Build time: Mon Jun 30 16:23:40 2025 (1751300620)
Build timestamp: 1751300620
Build timestamp as int: 1751300620
```

**Rules_python version:**

1.7.0

**Anything else relevant?**

Removing any support for "pynext" allows us to bump to 1.7.0. Additionally, bumping `PYNEXT_VERSION` to a different **minor** version than `PYTHON_VERSION` appears to work (eg current=3.13, next=3.14), but I can't fully confirm this yet because our requirements are not compatible with python 3.14.

This might be PEBKAC - if there's a more "correct" way of running multiple python versions please let me know. Our requirements for such are:

+ Do not build/test/run any "pynext" version by default
+ Only need to support versions N and N+1, where N might be a major, minor, or patch version
+ Must use the same requirements.txt lock file.
+ Must use the same pypi hub name
+ Or more accurately: the third-party deps of a target must not change between python versions.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.