bazel-contrib / bazel-contrib/rules_python

Packaging shared libraries with built py_binary

未关闭
#2,562 16 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Starlark
星标
688
派生
721
平均合并
15 小时 7 分钟
30 天内合并 PR
76

描述

I'm really sorry if this has been correctly answered somewhere else, I have looked at a lot of issues in this repo but haven't been able to come to a concrete solution.

So the question is pretty simple, I want to finally build my python binaries which are self runnable (no system dependencies, I plan to keep track of all the complicated packages and maintain this constraint). My end goal is to provide this behavior for both `bazel run` for development, and for the finally shipped built binary
This includes vendoring shared libraries which the targets depend on. I'm describing the issue I've faced in detail for a specific library, but the whole gist is that I want to provide deterministic ways for searching shared libraries (depending on different platforms, It can be env vars like `DYLD_LIBRARY_PATH`, or hardcoding the search path using @executable_path in linked python interpreter and dumping all libs in that path)

# The scenario
I have a very concrete scenario right now, installing [weasyprint](https://pypi.org/project/weasyprint/). I'll boil down the situation here:
- We need two system libraries, `libcairo.dylib` and `libgobject.dylib`. There might be more but I'm trying to figure this out with two of them first.
- The package loads it using `dlopen` system call
- This call searches `LD_LIBRARY_PATH` in linux
- `DYLD_LIBRARY_PATH` in Mac, and some predefined system folders
- Problem with Mac is, the OS by default removes `DYLD_LIBRARY_PATH` if its used to invoke a system interpreter (which happens cuz `py_binary` depends on system python)

Again, it's not just Mac that Im concerned about, but I want to provide users with all system packages (as much as I can) when they do a. bazel run. I also have an end goal of creating a self deployable packaged binary without any system dependencies for windows (without docker support)

# Stuff I tried

- I tried setting `DYLD_LIBRARY_PATH` in `py_binary.env` and it does not work, the generated python script for bootstrapping the process also doesn't seem to support this
- Also tried setting `DYLD_LIBRARY_PATH` in `main.py`, but it seems it should be set before launching for dlopen to use that path

## how conda does it
After some research, this is how Conda does it: https://docs.conda.io/projects/conda-build/en/stable/resources/use-shared-libraries.html#
This seems like a pretty good approach, in essence, they do this:
- set `@rpath` in the interpreter installed using Conda to point to `@loader_path/../lib/`
- put all shared libraries in libs folder
- conda controls the build process (which we do too kinda/sorta, I would rather build as less as I can, instead of starting by building an interpreter from scratch)
- Each built shared library sets `@rpath` to `@loader_path/`, thus making sure everyone tries to load everything from `lib` folder.

For windows they use `PATH` (slightly un-ideal but yea), linux seems to follow something similar

## patching the built interpreter
My next strategy was to try to replicate conda, but the packaged interpreter does not have @rpath set, patching it gives an error
```
$ install_name_tool -add_rpath @loader_path/. /Users/hariomnarang/Desktop/work/qureai/./bazel-bin/services/qure_platform_api/qure_platform_api.runfiles/python3_9_aarch64-apple-darwin/bin/python3

error: /Library/Developer/CommandLineTools/usr/bin/install_name_tool: changing install names or rpaths can't be redone for: /Users/hariomnarang/Desktop/work/qureai/./bazel-bin/services/qure_platform_api/qure_platform_api.runfiles/python3_9_aarch64-apple-darwin/bin/python3 (for architecture arm64) because larger updated load commands do not fit (the program must be relinked, and you may need to use -headerpad or -headerpad_max_install_names)
```
Now is there any way to do this other than creating my own toolchain rules? How does google do it?

Now I'm pretty stumped, all I can do right now, is just give users some directory containing all shared libs for some target and ask them to set it to a lib path search location, this is error prone and manual. `dlopen` and `ctypes.utils.find_library` are.

Thanks for the help

贡献指南

打开贡献指南

调研方向

该 issue 提到了 py_binary、其生成的 bootstrap 脚本、main.py 和自定义 toolchain 规则,但没有指出任何仓库文件或测试。首先追踪 py_binary 如何启动 Python 以及如何处理环境变量,然后定义一种可复现的方法,用于在不同平台上为 bazel run 和已发布的二进制文件打包并定位共享库。

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
build-system
Issue 类型
功能
难度
5/5
预计耗时
一周以上
活跃度
停滞
描述清晰度
需要澄清
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。