bazel-contrib / bazel-contrib/rules_python

py_zipapp duplicates shared libraries

Ouverte
#3,857 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
Starlark
Étoiles
688
Forks
721
Merge moyen
15 h 7 min
PR mergées (30 j)
76

Description

# 🐞 bug report

### Affected Rule

The issue is caused by the rule: `py_zipapp_binary`

With the following settings

```
common --@rules_python//python/config_settings:bootstrap_impl=script
common --@rules_python//python/config_settings:venvs_site_packages=yes
common --@rules_python//python/config_settings:venvs_use_declare_symlink=yes
```

### Is this a regression?

No

### Description

Shared libraries that shipped in pip packages are duplicated in the zipapp, once in the venv path and once in the wheel run files path. For native heavy packages, such as torch, this can significantly increase the size of the resulting zip file.

This is related to the closed #3439, but there the issue was packaging the run files using third-party rules.

The run file layout has both paths as symlinks pointing to the real file, but when building the zip file one is created as a `symlink` and the other as a `root_symlink`. These are added separately into the zip manifest and don't get deduped `tools/private/zipapp/zipper.py` and instead get byte-copied into the zip file.

I would expect these to be deduped, so that only one copied into the zip file and the other becomes a symlink to that.

## 🔬 Minimal Reproduction

```python
# BUILD.bazel
load("@rules_python//python:py_binary.bzl", "py_binary")
load("@rules_python//python:py_zipapp_binary.bzl", "py_zipapp_binary")

py_binary(
name = "demo",
srcs = ["demo.py"],
deps = ["@pypi//grpcio"],
)

py_zipapp_binary(name = "demo_zip", binary = ":demo")
```

```bash
$ bazel build //:demo_zip
$ unzip -l bazel-bin/demo_zip.pyz | grep cygrpc
2419632 1980-01-01 00:00 runfiles/rules_python++pip+pypi_311_grpcio_.../site-packages/grpc/_cython/cygrpc.cpython-311-x86_64-linux-gnu.so
2419632 1980-01-01 00:00 runfiles/_main//_demo.venv/lib/python3.11/site-packages/grpc/_cython/cygrpc.cpython-311-x86_64-linux-gnu.so
```

## 🔥 Exception or Error

No exceptions or errors

Guide de contribution

Ouvrir le guide de contribution

Piste de recherche

Commencez par tools/private/zipapp/zipper.py et la reproduction de py_zipapp_binary dans l’issue. Exécutez bazel build //:demo_zip, inspectez l’archive résultante avec unzip -l et comparez les deux chemins cygrpc. C’est terminé lorsque la shared library est copiée une seule fois et que l’autre chemin est représenté par un symlink.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
python
Domaine
build-system
Type d'issue
Bug
Difficulté
3/5
Temps estimé
1-2 jours
Activité
Calme
Clarté
Plutôt claire
Accessibilité débutants
68/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.