bazel-contrib / bazel-contrib/rules_python

Generate target_compatible_with for whl_library_targets

Đang mở
#3,686 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Starlark
Star
688
Fork
721
Merge trung bình
15 giờ 7 phút
Pull request đã merge (30 ngày)
76

Mô tả

# 🚀 feature request

### Relevant Rules

`whl_library_targets` macro in `python/private/pypi/whl_library_targets.bzl`

### Description

When `rules_python` generates BUILD targets for platform-specific wheels (e.g., `torch-2.8.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl`), the resulting `py_library` does **not** set `target_compatible_with`. This means Bazel cannot distinguish platform-specific wheels from pure-Python ones at analysis time.

We maintain a repo that targets both `x86_64` and `aarch64`. Our `pyproject.toml` uses PEP 508 environment markers to restrict certain dependencies to x86_64:

```toml
dependencies = [
"torch==2.8.0; platform_machine == 'x86_64'",
"opencv-python==4.11.0.86; platform_machine == 'x86_64'",
"tensorboard==2.20.0; platform_machine == 'x86_64'",
# ... many more x86_64-only deps
]
```

We generate separate lock files per platform via `uv pip compile`. The `pip.parse` extension correctly fetches platform-specific wheels. However, the generated `py_library` targets for these wheels have no `target_compatible_with` constraint.

This causes two issues:

1. **`bazel build //...` fails on aarch64**: Bazel attempts to build targets that transitively depend on x86_64-only wheels, which fail because the wheel files don't exist for aarch64. With `target_compatible_with`, Bazel would instead **skip** incompatible targets (printing `SKIPPED` in the build output).

2. **Manual `target_compatible_with` annotations required everywhere**: As a workaround, we must add `target_compatible_with = ["@platforms//cpu:x86_64"]` to every `py_test` and `py_binary` that transitively depends on an x86_64-only wheel. This is error-prone and tedious — our repo currently has dozens of such annotations scattered across BUILD files.

### Describe the solution you'd like

The `whl_library_targets` macro should automatically infer `target_compatible_with` from the wheel filename's platform tag and set it on the generated `py_library`.

Wheel filenames follow a well-defined format ([PEP 427](https://peps.python.org/pep-0427/#file-name-convention)): `{distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl`. The platform tag encodes the target architecture:

| Platform tag suffix | Constraint |
|---|---|
| `_x86_64`, `_amd64` | `@platforms//cpu:x86_64` |
| `_aarch64`, `_arm64` | `@platforms//cpu:aarch64` |
| `_i686` | `@platforms//cpu:x86_32` |
| `_ppc64le` | `@platforms//cpu:ppc` |
| `_s390x` | `@platforms//cpu:s390x` |
| `any` (pure-Python) | (no constraint) |

A possible implementation in `whl_library_targets.bzl`:

```starlark
load(":parse_whl_name.bzl", "parse_whl_name")

_PLATFORM_TAG_CPU_MAP = {
"x86_64": "@platforms//cpu:x86_64",
"amd64": "@platforms//cpu:x86_64",
"aarch64": "@platforms//cpu:aarch64",
"arm64": "@platforms//cpu:aarch64",
"i686": "@platforms//cpu:x86_32",
"ppc64le": "@platforms//cpu:ppc",
"s390x": "@platforms//cpu:s390x",
}

def _target_compatible_with_from_whl_name(whl_filename):
if not whl_filename or not whl_filename.endswith(".whl"):
return []
parsed = parse_whl_name(whl_filename)
if parsed.platform_tag == "any":
return []
for suffix, constraint in _PLATFORM_TAG_CPU_MAP.items():
if parsed.platform_tag.endswith("_" + suffix):
return [constraint]
return []
```

Then in the `whl_library_targets` function, pass it to `py_library`:

```starlark
rules.py_library(
...
target_compatible_with = _target_compatible_with_from_whl_name(name),
...
)
```

This leverages the existing `parse_whl_name` utility already available in `rules_python`.

### Describe alternatives you've considered

1. **Manual `target_compatible_with` on downstream targets**: This is what we currently do — every `py_test`/`py_binary` that transitively depends on an x86_64-only wheel gets a manual `target_compatible_with = ["@platforms//cpu:x86_64"]`. This is tedious, error-prone, and doesn't scale. Each new test file requires the developer to figure out whether any transitive dependency is platform-specific.

2. **Patching `rules_python` locally via `single_version_override`**: We currently apply a local patch to `whl_library_targets.bzl` (via `single_version_override` in `MODULE.bazel`) that implements the solution described above. This works but is a maintenance burden — we have to keep the patch in sync with upstream `rules_python` updates.

3. **Using `pip.parse` `whl_modifications` to post-hoc add `target_compatible_with`**: This would require listing every platform-specific wheel individually and is even more error-prone than option 1.

The information to infer `target_compatible_with` is already embedded in the wheel filename. Having `rules_python` set it automatically would eliminate all three workarounds and make cross-platform Bazel builds work correctly out of the box.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Bắt đầu trong python/private/pypi/whl_library_targets.bzl và kiểm tra tiện ích parse_whl_name được đề cập trong issue, theo dõi cách tên tệp wheel đi đến các target py_library được tạo. Công việc được xem là hoàn tất khi các thẻ wheel dành riêng cho nền tảng tạo ra ràng buộc target_compatible_with tương ứng, trong khi các wheel pure-Python vẫn không bị ràng buộc; hãy kiểm tra các target Bazel được tạo và hành vi build liên quan.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
python
Lĩnh vực
build-system
Loại issue
Tính năng
Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức độ hoạt động
Ít trao đổi
Độ rõ ràng
Khá rõ ràng
Mức phù hợp với người mới
52/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.