indygreg / indygreg/PyOxidizer
brotli
- Dominant language
- Rust
- Stars
- 6.2k
- Forks
- 256
- PR merge metrics
- No merged PRs in 30d
Description
Adding `brotlipy` fails looking for `-llibbrotli`, which is its own internal copy of the library.
```console
Running command git clone -q https://github.com/jayvdb/brotlipy /tmp/pip-req-build-0y42i1hu
Running command git submodule update --init --recursive -q
Compiling pyembed v0.4.0 (/path/to/project/brotli/pyembed)
Compiling brotli v0.1.0 (/path/to/project/brotli)
error: linking with `cc` failed: exit code: 1
|
= note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/brotli-f90a18d4864a2a16.15kbg8p4dc6ijdu1.rcgu.o" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/brotli-f90a18d4864a2a16.1jc1u2iuh94lh70o.rcgu.o" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/brotli-f90a18d4864a2a16.22q3drr76jbpis3w.rcgu.o" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/brotli-f90a18d4864a2a16.2syjn2icu1vwbxik.rcgu.o" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/brotli-f90a18d4864a2a16.3bu23526ltwdhtc7.rcgu.o" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/brotli-f90a18d4864a2a16.4y1e3ncj44u9ud8.rcgu.o" "-o" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/brotli-f90a18d4864a2a16" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/brotli-f90a18d4864a2a16.1fz3gb6hjy58le1w.rcgu.o" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-nodefaultlibs" "-L" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps" "-L" "/path/to/project/brotli/build/target/debug/deps" "-L" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/build/jemalloc-sys-6a2dcd7182e77aca/out/build/lib" "-L" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/pyoxidizer" "-L" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/libpyembed-f6d8ce5fcfbe2b59.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/libuuid-40f161259e10575f.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand-4587c77d50bf082f.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand_xorshift-e19a683b8360bb74.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand_pcg-161749b1ee75ec4d.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand_hc-90ed91a123f3b276.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand_chacha-bb755f08c6cd079a.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand_isaac-bcff91db4f001673.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand_core-2c32e48f67c62942.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand_os-47519e8a6d5b29ed.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand_jitter-4cb1ba9a013dd964.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand_core-3c552559ee9910d6.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/libjemalloc_sys-ceb86c614fe13a93.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/liblazy_static-03ee1ee8a2342a8e.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/libcpython-d6d9ce6fb32c53cc.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/libnum_traits-76cec740f4fe2b5d.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/libbyteorder-9bfb225a74a4f789.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/libpython3_sys-539fc76e435ddd51.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/liblibc-dae06a64a5f75d6d.rlib" "-Wl,--start-group" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-d655238cc7536e49.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-a1048e3b4b8487f7.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libbacktrace_sys-e303706b808d25cf.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-7c4c4a242f593b83.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-3a3b1d4df4be24e8.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-9782daa5de8380d8.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-58ece968a1125984.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-97498a9a22285b62.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-ecc3ad89a49704cf.rlib" "-Wl,--end-group" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-aeea5fc255482634.rlib" "-Wl,-Bdynamic" "-ldl" "-lm" "-llibbrotli" "-lstdc++" "-lpthread" "-lutil" "-lutil" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lutil"
= note: /usr/lib64/gcc/x86_64-suse-linux/9/../../../../x86_64-suse-linux/bin/ld: cannot find -llibbrotli
collect2: error: ld returned 1 exit status
error: aborting due to previous error
error: Could not compile `brotli`.
To learn more, run the command again with --verbose.
```
Using git+https://github.com/jayvdb/brotlipy@libbrotli-v1.0-support with envvar `USE_SHARED_BROTLI=1` works better, but then fails with
```py
>>> import brotli
Traceback (most recent call last):
File "", line 1, in
File "brotli", line 3, in
File "brotli.brotli", line 5, in
ModuleNotFoundError: No module named 'brotli._brotli'
```
The brotlipy package shadows the real package `Brotli` from Google, sharing the same installed package name. Luckily they share a similar Python API.
If I can install `brotlipy`, and then install `Brotli` afterwards, the result should be the real Brotli is embedded.
It looks like they are processed in the correct order
```
adding embedded module source: brotli
adding embedded module bytecode: brotli
adding embedded module source: brotli.brotli
adding embedded module bytecode: brotli.brotli
adding embedded module source: brotli.build
adding embedded module bytecode: brotli.build
adding embedded module source: brotli
adding embedded module bytecode: brotli
```
I know brotli.build only exists in brotlipy, so that was included first, and then hopefully overwritten by google Brotli.
It looks like the build is going well
```
resolving inputs for 3 built extension modules...
library brotlienc required by _brotli
library brotlidec required by _brotli
library stdc++ required by _brotli
library ffi required by _cffi_backend
```
I have a static libs build of brotli locally, but I do not see the two brotli static libs being included.
Despite that, it builds ok, but then the error is
```
File "/usr/lib/python3.7/site-packages/requests_mock/adapter.py", line 15, in
from requests.adapters import BaseAdapter
File "requests", line 43, in
File "urllib3", line 7, in
File "urllib3.connectionpool", line 29, in
File "urllib3.connection", line 40, in
File "urllib3.util", line 5, in
File "urllib3.util.request", line 9, in
File "brotli", line 12, in
AttributeError: module '_brotli' has no attribute '__version__'
```
Which looks like the real Google brotli was used, as it does the following during init,
```
import _brotli
# The library version.
__version__ = _brotli.__version__
```
But that suggests something is also wrong with the build of the real Google brotli library.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the `brotlipy` install and import failures, then inspect the `pyembed` and `brotli` build inputs and the logged `_brotli` dependencies (`brotlienc`, `brotlidec`, `stdc++`). Compare the embedded `brotli`, `brotli.brotli`, and `brotli.build` modules with Google Brotli's `_brotli`; done means the package installs and imports without the reported linker, missing-module, or `__version__` errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- build-system, tooling
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100