bazel-contrib / bazel-contrib/rules_python

Packaging shared libraries with built py_binary

Đang mở
#2,562 16 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Starlark
Star
688
Fork
721
Merge trung bình
15 giờ 7 phút
Pull request đã merge (30 ngày)
76

Mô tả

I'm really sorry if this has been correctly answered somewhere else, I have looked at a lot of issues in this repo but haven't been able to come to a concrete solution.

So the question is pretty simple, I want to finally build my python binaries which are self runnable (no system dependencies, I plan to keep track of all the complicated packages and maintain this constraint). My end goal is to provide this behavior for both `bazel run` for development, and for the finally shipped built binary
This includes vendoring shared libraries which the targets depend on. I'm describing the issue I've faced in detail for a specific library, but the whole gist is that I want to provide deterministic ways for searching shared libraries (depending on different platforms, It can be env vars like `DYLD_LIBRARY_PATH`, or hardcoding the search path using @executable_path in linked python interpreter and dumping all libs in that path)

# The scenario
I have a very concrete scenario right now, installing [weasyprint](https://pypi.org/project/weasyprint/). I'll boil down the situation here:
- We need two system libraries, `libcairo.dylib` and `libgobject.dylib`. There might be more but I'm trying to figure this out with two of them first.
- The package loads it using `dlopen` system call
- This call searches `LD_LIBRARY_PATH` in linux
- `DYLD_LIBRARY_PATH` in Mac, and some predefined system folders
- Problem with Mac is, the OS by default removes `DYLD_LIBRARY_PATH` if its used to invoke a system interpreter (which happens cuz `py_binary` depends on system python)

Again, it's not just Mac that Im concerned about, but I want to provide users with all system packages (as much as I can) when they do a. bazel run. I also have an end goal of creating a self deployable packaged binary without any system dependencies for windows (without docker support)

# Stuff I tried

- I tried setting `DYLD_LIBRARY_PATH` in `py_binary.env` and it does not work, the generated python script for bootstrapping the process also doesn't seem to support this
- Also tried setting `DYLD_LIBRARY_PATH` in `main.py`, but it seems it should be set before launching for dlopen to use that path

## how conda does it
After some research, this is how Conda does it: https://docs.conda.io/projects/conda-build/en/stable/resources/use-shared-libraries.html#
This seems like a pretty good approach, in essence, they do this:
- set `@rpath` in the interpreter installed using Conda to point to `@loader_path/../lib/`
- put all shared libraries in libs folder
- conda controls the build process (which we do too kinda/sorta, I would rather build as less as I can, instead of starting by building an interpreter from scratch)
- Each built shared library sets `@rpath` to `@loader_path/`, thus making sure everyone tries to load everything from `lib` folder.

For windows they use `PATH` (slightly un-ideal but yea), linux seems to follow something similar

## patching the built interpreter
My next strategy was to try to replicate conda, but the packaged interpreter does not have @rpath set, patching it gives an error
```
$ install_name_tool -add_rpath @loader_path/. /Users/hariomnarang/Desktop/work/qureai/./bazel-bin/services/qure_platform_api/qure_platform_api.runfiles/python3_9_aarch64-apple-darwin/bin/python3

error: /Library/Developer/CommandLineTools/usr/bin/install_name_tool: changing install names or rpaths can't be redone for: /Users/hariomnarang/Desktop/work/qureai/./bazel-bin/services/qure_platform_api/qure_platform_api.runfiles/python3_9_aarch64-apple-darwin/bin/python3 (for architecture arm64) because larger updated load commands do not fit (the program must be relinked, and you may need to use -headerpad or -headerpad_max_install_names)
```
Now is there any way to do this other than creating my own toolchain rules? How does google do it?

Now I'm pretty stumped, all I can do right now, is just give users some directory containing all shared libs for some target and ask them to set it to a lib path search location, this is error prone and manual. `dlopen` and `ctypes.utils.find_library` are.

Thanks for the help

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Issue nêu tên py_binary, bootstrap script được tạo ra của nó, main.py và các quy tắc toolchain tùy chỉnh, nhưng không có tệp hay bài kiểm thử nào trong repository. Hãy bắt đầu bằng cách lần theo cách py_binary khởi chạy Python và xử lý các biến môi trường, sau đó xác định một phương pháp có thể tái lập để đóng gói và định vị các thư viện dùng chung cho bazel run và các binary được phân phối trên nhiều nền tảng.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
python
Lĩnh vực
build-system
Loại issue
Tính năng
Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức độ hoạt động
Đình trệ
Độ rõ ràng
Cần làm rõ
Mức phù hợp với người mới
25/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.