bazel-contrib / bazel-contrib/rules_python

Generate target_compatible_with for whl_library_targets

Open
#3,686 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Starlark
Stars
688
Forks
721
Avg merge
14h 49m
Merged PRs (30d)
81

Description

# 🚀 feature request

### Relevant Rules

`whl_library_targets` macro in `python/private/pypi/whl_library_targets.bzl`

### Description

When `rules_python` generates BUILD targets for platform-specific wheels (e.g., `torch-2.8.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl`), the resulting `py_library` does **not** set `target_compatible_with`. This means Bazel cannot distinguish platform-specific wheels from pure-Python ones at analysis time.

We maintain a repo that targets both `x86_64` and `aarch64`. Our `pyproject.toml` uses PEP 508 environment markers to restrict certain dependencies to x86_64:

```toml
dependencies = [
"torch==2.8.0; platform_machine == 'x86_64'",
"opencv-python==4.11.0.86; platform_machine == 'x86_64'",
"tensorboard==2.20.0; platform_machine == 'x86_64'",
# ... many more x86_64-only deps
]
```

We generate separate lock files per platform via `uv pip compile`. The `pip.parse` extension correctly fetches platform-specific wheels. However, the generated `py_library` targets for these wheels have no `target_compatible_with` constraint.

This causes two issues:

1. **`bazel build //...` fails on aarch64**: Bazel attempts to build targets that transitively depend on x86_64-only wheels, which fail because the wheel files don't exist for aarch64. With `target_compatible_with`, Bazel would instead **skip** incompatible targets (printing `SKIPPED` in the build output).

2. **Manual `target_compatible_with` annotations required everywhere**: As a workaround, we must add `target_compatible_with = ["@platforms//cpu:x86_64"]` to every `py_test` and `py_binary` that transitively depends on an x86_64-only wheel. This is error-prone and tedious — our repo currently has dozens of such annotations scattered across BUILD files.

### Describe the solution you'd like

The `whl_library_targets` macro should automatically infer `target_compatible_with` from the wheel filename's platform tag and set it on the generated `py_library`.

Wheel filenames follow a well-defined format ([PEP 427](https://peps.python.org/pep-0427/#file-name-convention)): `{distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl`. The platform tag encodes the target architecture:

| Platform tag suffix | Constraint |
|---|---|
| `_x86_64`, `_amd64` | `@platforms//cpu:x86_64` |
| `_aarch64`, `_arm64` | `@platforms//cpu:aarch64` |
| `_i686` | `@platforms//cpu:x86_32` |
| `_ppc64le` | `@platforms//cpu:ppc` |
| `_s390x` | `@platforms//cpu:s390x` |
| `any` (pure-Python) | (no constraint) |

A possible implementation in `whl_library_targets.bzl`:

```starlark
load(":parse_whl_name.bzl", "parse_whl_name")

_PLATFORM_TAG_CPU_MAP = {
"x86_64": "@platforms//cpu:x86_64",
"amd64": "@platforms//cpu:x86_64",
"aarch64": "@platforms//cpu:aarch64",
"arm64": "@platforms//cpu:aarch64",
"i686": "@platforms//cpu:x86_32",
"ppc64le": "@platforms//cpu:ppc",
"s390x": "@platforms//cpu:s390x",
}

def _target_compatible_with_from_whl_name(whl_filename):
if not whl_filename or not whl_filename.endswith(".whl"):
return []
parsed = parse_whl_name(whl_filename)
if parsed.platform_tag == "any":
return []
for suffix, constraint in _PLATFORM_TAG_CPU_MAP.items():
if parsed.platform_tag.endswith("_" + suffix):
return [constraint]
return []
```

Then in the `whl_library_targets` function, pass it to `py_library`:

```starlark
rules.py_library(
...
target_compatible_with = _target_compatible_with_from_whl_name(name),
...
)
```

This leverages the existing `parse_whl_name` utility already available in `rules_python`.

### Describe alternatives you've considered

1. **Manual `target_compatible_with` on downstream targets**: This is what we currently do — every `py_test`/`py_binary` that transitively depends on an x86_64-only wheel gets a manual `target_compatible_with = ["@platforms//cpu:x86_64"]`. This is tedious, error-prone, and doesn't scale. Each new test file requires the developer to figure out whether any transitive dependency is platform-specific.

2. **Patching `rules_python` locally via `single_version_override`**: We currently apply a local patch to `whl_library_targets.bzl` (via `single_version_override` in `MODULE.bazel`) that implements the solution described above. This works but is a maintenance burden — we have to keep the patch in sync with upstream `rules_python` updates.

3. **Using `pip.parse` `whl_modifications` to post-hoc add `target_compatible_with`**: This would require listing every platform-specific wheel individually and is even more error-prone than option 1.

The information to infer `target_compatible_with` is already embedded in the wheel filename. Having `rules_python` set it automatically would eliminate all three workarounds and make cross-platform Bazel builds work correctly out of the box.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.