bazel-contrib / bazel-contrib/rules_python

py_zipapp duplicates shared libraries

未关闭
#3,857 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Starlark
星标
688
派生
721
平均合并
15 小时 7 分钟
30 天内合并 PR
76

描述

# 🐞 bug report

### Affected Rule

The issue is caused by the rule: `py_zipapp_binary`

With the following settings

```
common --@rules_python//python/config_settings:bootstrap_impl=script
common --@rules_python//python/config_settings:venvs_site_packages=yes
common --@rules_python//python/config_settings:venvs_use_declare_symlink=yes
```

### Is this a regression?

No

### Description

Shared libraries that shipped in pip packages are duplicated in the zipapp, once in the venv path and once in the wheel run files path. For native heavy packages, such as torch, this can significantly increase the size of the resulting zip file.

This is related to the closed #3439, but there the issue was packaging the run files using third-party rules.

The run file layout has both paths as symlinks pointing to the real file, but when building the zip file one is created as a `symlink` and the other as a `root_symlink`. These are added separately into the zip manifest and don't get deduped `tools/private/zipapp/zipper.py` and instead get byte-copied into the zip file.

I would expect these to be deduped, so that only one copied into the zip file and the other becomes a symlink to that.

## 🔬 Minimal Reproduction

```python
# BUILD.bazel
load("@rules_python//python:py_binary.bzl", "py_binary")
load("@rules_python//python:py_zipapp_binary.bzl", "py_zipapp_binary")

py_binary(
name = "demo",
srcs = ["demo.py"],
deps = ["@pypi//grpcio"],
)

py_zipapp_binary(name = "demo_zip", binary = ":demo")
```

```bash
$ bazel build //:demo_zip
$ unzip -l bazel-bin/demo_zip.pyz | grep cygrpc
2419632 1980-01-01 00:00 runfiles/rules_python++pip+pypi_311_grpcio_.../site-packages/grpc/_cython/cygrpc.cpython-311-x86_64-linux-gnu.so
2419632 1980-01-01 00:00 runfiles/_main//_demo.venv/lib/python3.11/site-packages/grpc/_cython/cygrpc.cpython-311-x86_64-linux-gnu.so
```

## 🔥 Exception or Error

No exceptions or errors

贡献指南

打开贡献指南

调研方向

从 tools/private/zipapp/zipper.py 和 issue 中的 py_zipapp_binary reproduction 开始。运行 bazel build //:demo_zip,使用 unzip -l 检查生成的归档,并比较两个 cygrpc 路径。当 shared library 只复制一次,另一个路径表示为 symlink 时,即视为完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
build-system
Issue 类型
缺陷
难度
3/5
预计耗时
1-2 天
活跃度
冷清
描述清晰度
基本清楚
新手友好度
68/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。