bazel-contrib / bazel-contrib/rules_python
Packages shadowed by other modules are no longer importable under bzlmod
- Dominant language
- Starlark
- Stars
- 688
- Forks
- 721
- Avg merge
- 14h 49m
- Merged PRs (30d)
- 81
Description
# 🐞 bug report
### Affected Rule
The issue is caused by the rule: `py_binary`/`py_library` under bzlmod
### Is this a regression?
Yes, this works under the WORKSPACE model. It is only bzlmod that is broken.
### Description
When two Bazel workspaces (with one depending on the other) both define packages with the same name (`common`, `core`, `helpers`, `private`, `util`, ...) the outer one will shadow the inner one. Without bzlmod this has always been solvable by using absolute imports with the workspace name as the prefix. However, under bzlmod, these absolute imports no longer work: when a module is used as an outer repository the prefix is `_main` but when a module is used as a dependency repository it is ``. And even if that was fine, there are further problems caused by `local_path_override` because it suffixes `~override` onto the folders in the runfiles directory.
From my perspective the best solution would be to make the generated `py_binary` wrapper script read the generated `_repo_mapping` file and set up the path such that absolute imports work the same way they've always worked. One way to make that happen would be to have the wrapper script create a temporary directory, create symlinks with the correct names pointing to the real folders, and then add the temporary directory to the PYTHONPATH. I could imagine there is probably some cleaner way I can't think of.
## 🔬 Minimal Reproduction
I put an example here: https://github.com/aschleck/bzlmod-shadow-python-problem (with another writeup in `README.md`.)
This repository illustrates three situations:
* [shadowed_with_workspace](https://github.com/aschleck/bzlmod-shadow-python-problem/tree/master/shadowed_with_workspace) shows a workspace depending on another workspace where both define a package named `core`. In this case, the `outer` workspace imports `dep` absolutely but `dep`'s import of itself fails due to being shadowed.
* [with_workspace](https://github.com/aschleck/bzlmod-shadow-python-problem/tree/master/with_workspace) illustrates the obvious fix: `dep` can always import itself with its workspace name as a prefix (like `import dep.core.`. Another solution would be to use relative imports, but those get unwieldy and annoying in large repos.
* [with_bzlmod](https://github.com/aschleck/bzlmod-shadow-python-problem/tree/master/with_bzlmod) fails due to `rules_python` not respecting the `_repo_mapping` file for imports. The absolute import of `dep` from `outer` fails because the folder in the runfiles is actually `dep~override` since `local_path_override` was used. Even if it was named `dep`, the code in `dep` itself could no longer assume that `import dep` refers to itself (because when its rules are run directly it is `_main`, and so it would need to do `import _main.core` not `import dep.core`.)
## 🔥 Exception or Error
The error is what you'd expect: the import fails.
~/bzlmod-shadow-python-problem/with_bzlmod/outer $ bazel run //core
INFO: Analyzed target //core:core (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //core:core up-to-date:
bazel-bin/core/core
INFO: Elapsed time: 0.212s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/core/core
imported outer.core.outer
Traceback (most recent call last):
File "/home/april/.cache/bazel/_bazel_april/1d28658223745047b82756f6e8e88a66/execroot/_main/bazel-out/k8-fastbuild/bin/core/core.runfiles/_main/core/outer.py", line 3, in
import dep.core.interface
ModuleNotFoundError: No module named 'dep'
## 🌍 Your Environment
**Operating System:**
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.2 LTS"
**Output of `bazel version`:**
Bazelisk version: v1.14.0
Build label: 6.1.1
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Wed Mar 15 15:44:56 2023 (1678895096)
Build timestamp: 1678895096
Build timestamp as int: 1678895096
**Rules_python version:**
0.20.0
**Anything else relevant?**
Thank you for maintaining these rules! It's been amazing to see how well this all works on the whole.
Contributor guide
Assessment
This issue has not been assessed yet.