bazel-contrib / bazel-contrib/rules_python

py_zipapp duplicates shared libraries

Offen
#3,857 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Starlark
Sterne
688
Forks
721
Ø Merge
15 Std. 7 Min.
Gemergte PRs (30 T.)
76

Beschreibung

# 🐞 bug report

### Affected Rule

The issue is caused by the rule: `py_zipapp_binary`

With the following settings

```
common --@rules_python//python/config_settings:bootstrap_impl=script
common --@rules_python//python/config_settings:venvs_site_packages=yes
common --@rules_python//python/config_settings:venvs_use_declare_symlink=yes
```

### Is this a regression?

No

### Description

Shared libraries that shipped in pip packages are duplicated in the zipapp, once in the venv path and once in the wheel run files path. For native heavy packages, such as torch, this can significantly increase the size of the resulting zip file.

This is related to the closed #3439, but there the issue was packaging the run files using third-party rules.

The run file layout has both paths as symlinks pointing to the real file, but when building the zip file one is created as a `symlink` and the other as a `root_symlink`. These are added separately into the zip manifest and don't get deduped `tools/private/zipapp/zipper.py` and instead get byte-copied into the zip file.

I would expect these to be deduped, so that only one copied into the zip file and the other becomes a symlink to that.

## 🔬 Minimal Reproduction

```python
# BUILD.bazel
load("@rules_python//python:py_binary.bzl", "py_binary")
load("@rules_python//python:py_zipapp_binary.bzl", "py_zipapp_binary")

py_binary(
name = "demo",
srcs = ["demo.py"],
deps = ["@pypi//grpcio"],
)

py_zipapp_binary(name = "demo_zip", binary = ":demo")
```

```bash
$ bazel build //:demo_zip
$ unzip -l bazel-bin/demo_zip.pyz | grep cygrpc
2419632 1980-01-01 00:00 runfiles/rules_python++pip+pypi_311_grpcio_.../site-packages/grpc/_cython/cygrpc.cpython-311-x86_64-linux-gnu.so
2419632 1980-01-01 00:00 runfiles/_main//_demo.venv/lib/python3.11/site-packages/grpc/_cython/cygrpc.cpython-311-x86_64-linux-gnu.so
```

## 🔥 Exception or Error

No exceptions or errors

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginne mit tools/private/zipapp/zipper.py und der py_zipapp_binary-Reproduktion im Issue. Führe bazel build //:demo_zip aus, untersuche das resultierende Archiv mit unzip -l und vergleiche die beiden cygrpc-Pfade. Als abgeschlossen gilt es, wenn die shared library einmal kopiert wird und der andere Pfad als symlink dargestellt wird.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python
Bereich
build-system
Issue-Typ
Bug
Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Aktivitätsstatus
Ruhig
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
68/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.