bazel-contrib / bazel-contrib/rules_python

Packaging shared libraries with built py_binary

Open
#2,562 16 comments 0 reactions 0 assignees View on GitHub
Dominant language
Starlark
Stars
688
Forks
721
Avg merge
14h 49m
Merged PRs (30d)
81

Description

I'm really sorry if this has been correctly answered somewhere else, I have looked at a lot of issues in this repo but haven't been able to come to a concrete solution.

So the question is pretty simple, I want to finally build my python binaries which are self runnable (no system dependencies, I plan to keep track of all the complicated packages and maintain this constraint). My end goal is to provide this behavior for both `bazel run` for development, and for the finally shipped built binary
This includes vendoring shared libraries which the targets depend on. I'm describing the issue I've faced in detail for a specific library, but the whole gist is that I want to provide deterministic ways for searching shared libraries (depending on different platforms, It can be env vars like `DYLD_LIBRARY_PATH`, or hardcoding the search path using @executable_path in linked python interpreter and dumping all libs in that path)

# The scenario
I have a very concrete scenario right now, installing [weasyprint](https://pypi.org/project/weasyprint/). I'll boil down the situation here:
- We need two system libraries, `libcairo.dylib` and `libgobject.dylib`. There might be more but I'm trying to figure this out with two of them first.
- The package loads it using `dlopen` system call
- This call searches `LD_LIBRARY_PATH` in linux
- `DYLD_LIBRARY_PATH` in Mac, and some predefined system folders
- Problem with Mac is, the OS by default removes `DYLD_LIBRARY_PATH` if its used to invoke a system interpreter (which happens cuz `py_binary` depends on system python)

Again, it's not just Mac that Im concerned about, but I want to provide users with all system packages (as much as I can) when they do a. bazel run. I also have an end goal of creating a self deployable packaged binary without any system dependencies for windows (without docker support)

# Stuff I tried

- I tried setting `DYLD_LIBRARY_PATH` in `py_binary.env` and it does not work, the generated python script for bootstrapping the process also doesn't seem to support this
- Also tried setting `DYLD_LIBRARY_PATH` in `main.py`, but it seems it should be set before launching for dlopen to use that path

## how conda does it
After some research, this is how Conda does it: https://docs.conda.io/projects/conda-build/en/stable/resources/use-shared-libraries.html#
This seems like a pretty good approach, in essence, they do this:
- set `@rpath` in the interpreter installed using Conda to point to `@loader_path/../lib/`
- put all shared libraries in libs folder
- conda controls the build process (which we do too kinda/sorta, I would rather build as less as I can, instead of starting by building an interpreter from scratch)
- Each built shared library sets `@rpath` to `@loader_path/`, thus making sure everyone tries to load everything from `lib` folder.

For windows they use `PATH` (slightly un-ideal but yea), linux seems to follow something similar

## patching the built interpreter
My next strategy was to try to replicate conda, but the packaged interpreter does not have @rpath set, patching it gives an error
```
$ install_name_tool -add_rpath @loader_path/. /Users/hariomnarang/Desktop/work/qureai/./bazel-bin/services/qure_platform_api/qure_platform_api.runfiles/python3_9_aarch64-apple-darwin/bin/python3

error: /Library/Developer/CommandLineTools/usr/bin/install_name_tool: changing install names or rpaths can't be redone for: /Users/hariomnarang/Desktop/work/qureai/./bazel-bin/services/qure_platform_api/qure_platform_api.runfiles/python3_9_aarch64-apple-darwin/bin/python3 (for architecture arm64) because larger updated load commands do not fit (the program must be relinked, and you may need to use -headerpad or -headerpad_max_install_names)
```
Now is there any way to do this other than creating my own toolchain rules? How does google do it?

Now I'm pretty stumped, all I can do right now, is just give users some directory containing all shared libs for some target and ask them to set it to a lib path search location, this is error prone and manual. `dlopen` and `ctypes.utils.find_library` are.

Thanks for the help

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.