bazel-contrib / bazel-contrib/rules_python
py_zipapp duplicates shared libraries
- Lingua principale
- Starlark
- Stelle
- 688
- Fork
- 721
- Merge medio
- 15h 7m
- PR unite (30g)
- 76
Descrizione
# 🐞 bug report
### Affected Rule
The issue is caused by the rule: `py_zipapp_binary`
With the following settings
```
common --@rules_python//python/config_settings:bootstrap_impl=script
common --@rules_python//python/config_settings:venvs_site_packages=yes
common --@rules_python//python/config_settings:venvs_use_declare_symlink=yes
```
### Is this a regression?
No
### Description
Shared libraries that shipped in pip packages are duplicated in the zipapp, once in the venv path and once in the wheel run files path. For native heavy packages, such as torch, this can significantly increase the size of the resulting zip file.
This is related to the closed #3439, but there the issue was packaging the run files using third-party rules.
The run file layout has both paths as symlinks pointing to the real file, but when building the zip file one is created as a `symlink` and the other as a `root_symlink`. These are added separately into the zip manifest and don't get deduped `tools/private/zipapp/zipper.py` and instead get byte-copied into the zip file.
I would expect these to be deduped, so that only one copied into the zip file and the other becomes a symlink to that.
## 🔬 Minimal Reproduction
```python
# BUILD.bazel
load("@rules_python//python:py_binary.bzl", "py_binary")
load("@rules_python//python:py_zipapp_binary.bzl", "py_zipapp_binary")
py_binary(
name = "demo",
srcs = ["demo.py"],
deps = ["@pypi//grpcio"],
)
py_zipapp_binary(name = "demo_zip", binary = ":demo")
```
```bash
$ bazel build //:demo_zip
$ unzip -l bazel-bin/demo_zip.pyz | grep cygrpc
2419632 1980-01-01 00:00 runfiles/rules_python++pip+pypi_311_grpcio_.../site-packages/grpc/_cython/cygrpc.cpython-311-x86_64-linux-gnu.so
2419632 1980-01-01 00:00 runfiles/_main//_demo.venv/lib/python3.11/site-packages/grpc/_cython/cygrpc.cpython-311-x86_64-linux-gnu.so
```
## 🔥 Exception or Error
No exceptions or errors
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Start with tools/private/zipapp/zipper.py and the py_zipapp_binary reproduction in the issue. Run bazel build //:demo_zip, inspect the resulting archive with unzip -l and compare the two cygrpc paths. Done means the shared library is copied once and the other path is represented as a symlink.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- python
- Ambito
- build-system
- Tipo di issue
- Bug
- Difficoltà
- 3/5
- Tempo stimato
- 1-2 giorni
- Stato di attività
- Tranquilla
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 68/100