bazel-contrib / bazel-contrib/rules_python
Packaging shared libraries with built py_binary
- 主要言語
- Starlark
- スター
- 688
- フォーク
- 721
- 平均マージ
- 15時間 7分
- マージ済み PR(30日)
- 76
説明
I'm really sorry if this has been correctly answered somewhere else, I have looked at a lot of issues in this repo but haven't been able to come to a concrete solution.
So the question is pretty simple, I want to finally build my python binaries which are self runnable (no system dependencies, I plan to keep track of all the complicated packages and maintain this constraint). My end goal is to provide this behavior for both `bazel run` for development, and for the finally shipped built binary
This includes vendoring shared libraries which the targets depend on. I'm describing the issue I've faced in detail for a specific library, but the whole gist is that I want to provide deterministic ways for searching shared libraries (depending on different platforms, It can be env vars like `DYLD_LIBRARY_PATH`, or hardcoding the search path using @executable_path in linked python interpreter and dumping all libs in that path)
# The scenario
I have a very concrete scenario right now, installing [weasyprint](https://pypi.org/project/weasyprint/). I'll boil down the situation here:
- We need two system libraries, `libcairo.dylib` and `libgobject.dylib`. There might be more but I'm trying to figure this out with two of them first.
- The package loads it using `dlopen` system call
- This call searches `LD_LIBRARY_PATH` in linux
- `DYLD_LIBRARY_PATH` in Mac, and some predefined system folders
- Problem with Mac is, the OS by default removes `DYLD_LIBRARY_PATH` if its used to invoke a system interpreter (which happens cuz `py_binary` depends on system python)
Again, it's not just Mac that Im concerned about, but I want to provide users with all system packages (as much as I can) when they do a. bazel run. I also have an end goal of creating a self deployable packaged binary without any system dependencies for windows (without docker support)
# Stuff I tried
- I tried setting `DYLD_LIBRARY_PATH` in `py_binary.env` and it does not work, the generated python script for bootstrapping the process also doesn't seem to support this
- Also tried setting `DYLD_LIBRARY_PATH` in `main.py`, but it seems it should be set before launching for dlopen to use that path
## how conda does it
After some research, this is how Conda does it: https://docs.conda.io/projects/conda-build/en/stable/resources/use-shared-libraries.html#
This seems like a pretty good approach, in essence, they do this:
- set `@rpath` in the interpreter installed using Conda to point to `@loader_path/../lib/`
- put all shared libraries in libs folder
- conda controls the build process (which we do too kinda/sorta, I would rather build as less as I can, instead of starting by building an interpreter from scratch)
- Each built shared library sets `@rpath` to `@loader_path/`, thus making sure everyone tries to load everything from `lib` folder.
For windows they use `PATH` (slightly un-ideal but yea), linux seems to follow something similar
## patching the built interpreter
My next strategy was to try to replicate conda, but the packaged interpreter does not have @rpath set, patching it gives an error
```
$ install_name_tool -add_rpath @loader_path/. /Users/hariomnarang/Desktop/work/qureai/./bazel-bin/services/qure_platform_api/qure_platform_api.runfiles/python3_9_aarch64-apple-darwin/bin/python3
error: /Library/Developer/CommandLineTools/usr/bin/install_name_tool: changing install names or rpaths can't be redone for: /Users/hariomnarang/Desktop/work/qureai/./bazel-bin/services/qure_platform_api/qure_platform_api.runfiles/python3_9_aarch64-apple-darwin/bin/python3 (for architecture arm64) because larger updated load commands do not fit (the program must be relinked, and you may need to use -headerpad or -headerpad_max_install_names)
```
Now is there any way to do this other than creating my own toolchain rules? How does google do it?
Now I'm pretty stumped, all I can do right now, is just give users some directory containing all shared libs for some target and ask them to set it to a lib path search location, this is error prone and manual. `dlopen` and `ctypes.utils.find_library` are.
Thanks for the help
コントリビューションガイド
調査の方向性
この issue では、py_binary、その生成された bootstrap スクリプト、main.py、カスタム toolchain ルールが挙げられていますが、リポジトリ内のファイルやテストは挙げられていません。まず、py_binary が Python を起動し、環境変数を処理する方法を追跡し、その後、各プラットフォームで bazel run と配布バイナリ向けに共有ライブラリをバンドルして見つけるための再現可能な方法を定義します。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- python
- 領域
- build-system
- issue の種類
- 機能追加
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 活発さ
- 停滞
- 明瞭さ
- 説明が足りない
- 初心者へのやさしさ
- 25/100