bazel-contrib / bazel-contrib/rules_python
Generate target_compatible_with for whl_library_targets
- Dominant language
- Starlark
- Stars
- 688
- Forks
- 721
- Avg merge
- 14h 49m
- Merged PRs (30d)
- 81
Description
# 🚀 feature request
### Relevant Rules
`whl_library_targets` macro in `python/private/pypi/whl_library_targets.bzl`
### Description
When `rules_python` generates BUILD targets for platform-specific wheels (e.g., `torch-2.8.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl`), the resulting `py_library` does **not** set `target_compatible_with`. This means Bazel cannot distinguish platform-specific wheels from pure-Python ones at analysis time.
We maintain a repo that targets both `x86_64` and `aarch64`. Our `pyproject.toml` uses PEP 508 environment markers to restrict certain dependencies to x86_64:
```toml
dependencies = [
"torch==2.8.0; platform_machine == 'x86_64'",
"opencv-python==4.11.0.86; platform_machine == 'x86_64'",
"tensorboard==2.20.0; platform_machine == 'x86_64'",
# ... many more x86_64-only deps
]
```
We generate separate lock files per platform via `uv pip compile`. The `pip.parse` extension correctly fetches platform-specific wheels. However, the generated `py_library` targets for these wheels have no `target_compatible_with` constraint.
This causes two issues:
1. **`bazel build //...` fails on aarch64**: Bazel attempts to build targets that transitively depend on x86_64-only wheels, which fail because the wheel files don't exist for aarch64. With `target_compatible_with`, Bazel would instead **skip** incompatible targets (printing `SKIPPED` in the build output).
2. **Manual `target_compatible_with` annotations required everywhere**: As a workaround, we must add `target_compatible_with = ["@platforms//cpu:x86_64"]` to every `py_test` and `py_binary` that transitively depends on an x86_64-only wheel. This is error-prone and tedious — our repo currently has dozens of such annotations scattered across BUILD files.
### Describe the solution you'd like
The `whl_library_targets` macro should automatically infer `target_compatible_with` from the wheel filename's platform tag and set it on the generated `py_library`.
Wheel filenames follow a well-defined format ([PEP 427](https://peps.python.org/pep-0427/#file-name-convention)): `{distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl`. The platform tag encodes the target architecture:
| Platform tag suffix | Constraint |
|---|---|
| `_x86_64`, `_amd64` | `@platforms//cpu:x86_64` |
| `_aarch64`, `_arm64` | `@platforms//cpu:aarch64` |
| `_i686` | `@platforms//cpu:x86_32` |
| `_ppc64le` | `@platforms//cpu:ppc` |
| `_s390x` | `@platforms//cpu:s390x` |
| `any` (pure-Python) | (no constraint) |
A possible implementation in `whl_library_targets.bzl`:
```starlark
load(":parse_whl_name.bzl", "parse_whl_name")
_PLATFORM_TAG_CPU_MAP = {
"x86_64": "@platforms//cpu:x86_64",
"amd64": "@platforms//cpu:x86_64",
"aarch64": "@platforms//cpu:aarch64",
"arm64": "@platforms//cpu:aarch64",
"i686": "@platforms//cpu:x86_32",
"ppc64le": "@platforms//cpu:ppc",
"s390x": "@platforms//cpu:s390x",
}
def _target_compatible_with_from_whl_name(whl_filename):
if not whl_filename or not whl_filename.endswith(".whl"):
return []
parsed = parse_whl_name(whl_filename)
if parsed.platform_tag == "any":
return []
for suffix, constraint in _PLATFORM_TAG_CPU_MAP.items():
if parsed.platform_tag.endswith("_" + suffix):
return [constraint]
return []
```
Then in the `whl_library_targets` function, pass it to `py_library`:
```starlark
rules.py_library(
...
target_compatible_with = _target_compatible_with_from_whl_name(name),
...
)
```
This leverages the existing `parse_whl_name` utility already available in `rules_python`.
### Describe alternatives you've considered
1. **Manual `target_compatible_with` on downstream targets**: This is what we currently do — every `py_test`/`py_binary` that transitively depends on an x86_64-only wheel gets a manual `target_compatible_with = ["@platforms//cpu:x86_64"]`. This is tedious, error-prone, and doesn't scale. Each new test file requires the developer to figure out whether any transitive dependency is platform-specific.
2. **Patching `rules_python` locally via `single_version_override`**: We currently apply a local patch to `whl_library_targets.bzl` (via `single_version_override` in `MODULE.bazel`) that implements the solution described above. This works but is a maintenance burden — we have to keep the patch in sync with upstream `rules_python` updates.
3. **Using `pip.parse` `whl_modifications` to post-hoc add `target_compatible_with`**: This would require listing every platform-specific wheel individually and is even more error-prone than option 1.
The information to infer `target_compatible_with` is already embedded in the wheel filename. Having `rules_python` set it automatically would eliminate all three workarounds and make cross-platform Bazel builds work correctly out of the box.
Contributor guide
Assessment
This issue has not been assessed yet.