indygreg / indygreg/PyOxidizer

Shared lib builder drops package directory

Open
#169 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
6.2k
Forks
256
PR merge metrics
No merged PRs in 30d

Description

Shared libs that are in packages that also contain python code are stored in `site-packages/foo/bar.so`

PyOxidizer is dropping the `foo` package name, creating conflicts.

A very prominent example is `markupsafe`, `simplejson` and `logbook` all have a `_speedups.so`

```py
>>> import simplejson
import simplejson as json
File "simplejson", line 115, in
File "simplejson.encoder", line 15, in
File "simplejson.encoder", line 12, in _import_speedups
AttributeError: module 'markupsafe._speedups' has no attribute 'encode_basestring_ascii'
```

`tornado` and `websockets` and `fastparquet` all have a `speedups.so`

`cytoolz` and `h5py` and `espressomd` all have `utils.so`, and further nested are `zmq/backend/cython/utils.so` and `yt/analysis_modules/photon_simulator/utils.so`

`dulwich` and `h5py` have `_objects.so`

`xxhash` and `lmdb` have `cpython.so`

`bsdiff4` and `espeak` have `core.so`

`astropy` and `sunpy` have `_compiler.so`

`gevent` and `kivy` both have `_event.so`

PyQt4 and PyQt5 have an almost identical set of built modules under their respective directories.

Cryptodome and Crypto also share a set of similar names.

Third level objects `sklearn/manifold/_utils.so` and `sklearn/tree/_utils.so` probably also have this problem, breaking sklearn even if only it is installed.

Third level objects `gssapi/raw/misc.so` and `samba/dcerpc/misc.so` likely also conflict.

Third level `astropy/stats/_stats.so` and `scipy/stats/_stats.so`

As noted at https://github.com/indygreg/PyOxidizer/issues/167 , brotli and brotlipy both expect `from brotli import _brotli` to import their .so , but that example is a bit messy as the two libraries conflict anyway.

And that is just what I can find in my site-packages :P Obviously there will be many more conflicts I am unaware of.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the collision with packages such as markupsafe, simplejson, and logbook, then trace the shared lib builder responsible for paths under site-packages. Verify that nested package directories are preserved so same-named .so files remain importable without collisions; the issue names no test or source file, so repository-wide search is required.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
build-system
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.