indygreg / indygreg/PyOxidizer

brotli

Open
#167 4 comments 0 reactions 0 assignees View on GitHub
compatibility
Dominant language
Rust
Stars
6.2k
Forks
256
PR merge metrics
No merged PRs in 30d

Description

Adding `brotlipy` fails looking for `-llibbrotli`, which is its own internal copy of the library.

```console
Running command git clone -q https://github.com/jayvdb/brotlipy /tmp/pip-req-build-0y42i1hu
Running command git submodule update --init --recursive -q
Compiling pyembed v0.4.0 (/path/to/project/brotli/pyembed)
Compiling brotli v0.1.0 (/path/to/project/brotli)
error: linking with `cc` failed: exit code: 1
|
= note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/brotli-f90a18d4864a2a16.15kbg8p4dc6ijdu1.rcgu.o" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/brotli-f90a18d4864a2a16.1jc1u2iuh94lh70o.rcgu.o" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/brotli-f90a18d4864a2a16.22q3drr76jbpis3w.rcgu.o" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/brotli-f90a18d4864a2a16.2syjn2icu1vwbxik.rcgu.o" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/brotli-f90a18d4864a2a16.3bu23526ltwdhtc7.rcgu.o" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/brotli-f90a18d4864a2a16.4y1e3ncj44u9ud8.rcgu.o" "-o" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/brotli-f90a18d4864a2a16" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/brotli-f90a18d4864a2a16.1fz3gb6hjy58le1w.rcgu.o" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-nodefaultlibs" "-L" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps" "-L" "/path/to/project/brotli/build/target/debug/deps" "-L" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/build/jemalloc-sys-6a2dcd7182e77aca/out/build/lib" "-L" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/pyoxidizer" "-L" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/libpyembed-f6d8ce5fcfbe2b59.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/libuuid-40f161259e10575f.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand-4587c77d50bf082f.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand_xorshift-e19a683b8360bb74.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand_pcg-161749b1ee75ec4d.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand_hc-90ed91a123f3b276.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand_chacha-bb755f08c6cd079a.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand_isaac-bcff91db4f001673.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand_core-2c32e48f67c62942.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand_os-47519e8a6d5b29ed.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand_jitter-4cb1ba9a013dd964.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand_core-3c552559ee9910d6.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/libjemalloc_sys-ceb86c614fe13a93.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/liblazy_static-03ee1ee8a2342a8e.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/libcpython-d6d9ce6fb32c53cc.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/libnum_traits-76cec740f4fe2b5d.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/libbyteorder-9bfb225a74a4f789.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/libpython3_sys-539fc76e435ddd51.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/liblibc-dae06a64a5f75d6d.rlib" "-Wl,--start-group" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-d655238cc7536e49.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-a1048e3b4b8487f7.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libbacktrace_sys-e303706b808d25cf.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-7c4c4a242f593b83.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-3a3b1d4df4be24e8.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-9782daa5de8380d8.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-58ece968a1125984.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-97498a9a22285b62.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-ecc3ad89a49704cf.rlib" "-Wl,--end-group" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-aeea5fc255482634.rlib" "-Wl,-Bdynamic" "-ldl" "-lm" "-llibbrotli" "-lstdc++" "-lpthread" "-lutil" "-lutil" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lutil"
= note: /usr/lib64/gcc/x86_64-suse-linux/9/../../../../x86_64-suse-linux/bin/ld: cannot find -llibbrotli
collect2: error: ld returned 1 exit status

error: aborting due to previous error

error: Could not compile `brotli`.

To learn more, run the command again with --verbose.
```

Using git+https://github.com/jayvdb/brotlipy@libbrotli-v1.0-support with envvar `USE_SHARED_BROTLI=1` works better, but then fails with

```py
>>> import brotli
Traceback (most recent call last):
File "", line 1, in
File "brotli", line 3, in
File "brotli.brotli", line 5, in
ModuleNotFoundError: No module named 'brotli._brotli'
```

The brotlipy package shadows the real package `Brotli` from Google, sharing the same installed package name. Luckily they share a similar Python API.
If I can install `brotlipy`, and then install `Brotli` afterwards, the result should be the real Brotli is embedded.

It looks like they are processed in the correct order
```
adding embedded module source: brotli
adding embedded module bytecode: brotli
adding embedded module source: brotli.brotli
adding embedded module bytecode: brotli.brotli
adding embedded module source: brotli.build
adding embedded module bytecode: brotli.build
adding embedded module source: brotli
adding embedded module bytecode: brotli
```
I know brotli.build only exists in brotlipy, so that was included first, and then hopefully overwritten by google Brotli.

It looks like the build is going well

```
resolving inputs for 3 built extension modules...
library brotlienc required by _brotli
library brotlidec required by _brotli
library stdc++ required by _brotli
library ffi required by _cffi_backend
```
I have a static libs build of brotli locally, but I do not see the two brotli static libs being included.

Despite that, it builds ok, but then the error is

```
File "/usr/lib/python3.7/site-packages/requests_mock/adapter.py", line 15, in
from requests.adapters import BaseAdapter
File "requests", line 43, in
File "urllib3", line 7, in
File "urllib3.connectionpool", line 29, in
File "urllib3.connection", line 40, in
File "urllib3.util", line 5, in
File "urllib3.util.request", line 9, in
File "brotli", line 12, in
AttributeError: module '_brotli' has no attribute '__version__'
```
Which looks like the real Google brotli was used, as it does the following during init,
```
import _brotli

# The library version.
__version__ = _brotli.__version__
```
But that suggests something is also wrong with the build of the real Google brotli library.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the `brotlipy` install and import failures, then inspect the `pyembed` and `brotli` build inputs and the logged `_brotli` dependencies (`brotlienc`, `brotlidec`, `stdc++`). Compare the embedded `brotli`, `brotli.brotli`, and `brotli.build` modules with Google Brotli's `_brotli`; done means the package installs and imports without the reported linker, missing-module, or `__version__` errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
build-system, tooling
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.