bazel-contrib / bazel-contrib/rules_python

Packaging shared libraries with built py_binary

Ouverte
#2,562 16 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
Starlark
Étoiles
688
Forks
721
Merge moyen
15 h 7 min
PR mergées (30 j)
76

Description

I'm really sorry if this has been correctly answered somewhere else, I have looked at a lot of issues in this repo but haven't been able to come to a concrete solution.

So the question is pretty simple, I want to finally build my python binaries which are self runnable (no system dependencies, I plan to keep track of all the complicated packages and maintain this constraint). My end goal is to provide this behavior for both `bazel run` for development, and for the finally shipped built binary
This includes vendoring shared libraries which the targets depend on. I'm describing the issue I've faced in detail for a specific library, but the whole gist is that I want to provide deterministic ways for searching shared libraries (depending on different platforms, It can be env vars like `DYLD_LIBRARY_PATH`, or hardcoding the search path using @executable_path in linked python interpreter and dumping all libs in that path)

# The scenario
I have a very concrete scenario right now, installing [weasyprint](https://pypi.org/project/weasyprint/). I'll boil down the situation here:
- We need two system libraries, `libcairo.dylib` and `libgobject.dylib`. There might be more but I'm trying to figure this out with two of them first.
- The package loads it using `dlopen` system call
- This call searches `LD_LIBRARY_PATH` in linux
- `DYLD_LIBRARY_PATH` in Mac, and some predefined system folders
- Problem with Mac is, the OS by default removes `DYLD_LIBRARY_PATH` if its used to invoke a system interpreter (which happens cuz `py_binary` depends on system python)

Again, it's not just Mac that Im concerned about, but I want to provide users with all system packages (as much as I can) when they do a. bazel run. I also have an end goal of creating a self deployable packaged binary without any system dependencies for windows (without docker support)

# Stuff I tried

- I tried setting `DYLD_LIBRARY_PATH` in `py_binary.env` and it does not work, the generated python script for bootstrapping the process also doesn't seem to support this
- Also tried setting `DYLD_LIBRARY_PATH` in `main.py`, but it seems it should be set before launching for dlopen to use that path

## how conda does it
After some research, this is how Conda does it: https://docs.conda.io/projects/conda-build/en/stable/resources/use-shared-libraries.html#
This seems like a pretty good approach, in essence, they do this:
- set `@rpath` in the interpreter installed using Conda to point to `@loader_path/../lib/`
- put all shared libraries in libs folder
- conda controls the build process (which we do too kinda/sorta, I would rather build as less as I can, instead of starting by building an interpreter from scratch)
- Each built shared library sets `@rpath` to `@loader_path/`, thus making sure everyone tries to load everything from `lib` folder.

For windows they use `PATH` (slightly un-ideal but yea), linux seems to follow something similar

## patching the built interpreter
My next strategy was to try to replicate conda, but the packaged interpreter does not have @rpath set, patching it gives an error
```
$ install_name_tool -add_rpath @loader_path/. /Users/hariomnarang/Desktop/work/qureai/./bazel-bin/services/qure_platform_api/qure_platform_api.runfiles/python3_9_aarch64-apple-darwin/bin/python3

error: /Library/Developer/CommandLineTools/usr/bin/install_name_tool: changing install names or rpaths can't be redone for: /Users/hariomnarang/Desktop/work/qureai/./bazel-bin/services/qure_platform_api/qure_platform_api.runfiles/python3_9_aarch64-apple-darwin/bin/python3 (for architecture arm64) because larger updated load commands do not fit (the program must be relinked, and you may need to use -headerpad or -headerpad_max_install_names)
```
Now is there any way to do this other than creating my own toolchain rules? How does google do it?

Now I'm pretty stumped, all I can do right now, is just give users some directory containing all shared libs for some target and ask them to set it to a lib path search location, this is error prone and manual. `dlopen` and `ctypes.utils.find_library` are.

Thanks for the help

Guide de contribution

Ouvrir le guide de contribution

Piste de recherche

L’issue mentionne py_binary, son script bootstrap généré, main.py et des règles de toolchain personnalisées, mais aucun fichier ni test du dépôt. Commencez par retracer comment py_binary lance Python et gère les variables d’environnement, puis définissez une approche reproductible pour regrouper et localiser les bibliothèques partagées pour bazel run et les binaires distribués sur différentes plateformes.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
python
Domaine
build-system
Type d'issue
Fonctionnalité
Difficulté
5/5
Temps estimé
Plus d'une semaine
Activité
À l'abandon
Clarté
À clarifier
Accessibilité débutants
25/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.