indygreg / indygreg/PyOxidizer

Incorrect copied shared library name in filesystem relative resource policy

Open
#284 10 comments 0 reactions 0 assignees View on GitHub
bug need-info
Dominant language
Rust
Stars
6.2k
Forks
256
PR merge metrics
No merged PRs in 30d

Description

When trying to use 0.7.0 to build a basic app with only numpy added, the new `prefer-in-memory-fallback-filesystem-relative` resource policy can be used. This almost works, but one of the shared libraries is copied into the incorrect location in the specified folder.

The `pyoxidizer.bzl` file is effectively the default generated file from the getting started guide, but with the following lines:

- `resources_policy='prefer-in-memory-fallback-filesystem-relative:lib',`
- `exe.add_filesystem_relative_python_resources('lib', dist.pip_install(['numpy']))`

When running the binary, then doing `import numpy`, the following (trimmed) error occurs:

`Original error was: libopenblasp-r0-ae94cfde.3.9.dev.so: cannot open shared object file: No such file or directory`

When looking at the file tree alongside the binary, there is the file at `build/x86_64-unknown-linux-gnu/debug/install/lib/numpy/libs/libopenblasp-r0-ae94cfde/3/9/dev.so`, which should instead be at `build/x86_64-unknown-linux-gnu/debug/install/lib/numpy.libs/libopenblasp-r0-ae94cfde.3.9.dev.so`

It appears `.` in the file name (`libopenblasp-r0-ae94cfde.3.9.dev.so`) and folder name (`numpy.libs`) have been converted to `/`. If you look in the numpy wheel, you can see the expected file tree structure. Having a vague dig through the PyOxidizer code, I can't see why this one file is getting mis-copied, particularly when there are three other shared libraries in that folder which are copied correctly. The only difference is the order of the version numbers around the `.so`. Is the shared lib path perhaps getting interpretted as a python module name?

If I manually move that file to the `numpy.libs` folder, numpy now imports correctly and works for the few things I've tested. If this can be resolved, numpy will effectively work out-of-the-box and resolve #65 .

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the issue with the generated pyoxidizer.bzl configuration, numpy, and the prefer-in-memory-fallback-filesystem-relative policy. Compare the NumPy wheel tree with the generated build/x86_64-unknown-linux-gnu/debug/install/lib paths, focusing on how the shared-library and numpy.libs names are copied. Done means the library remains at lib/numpy.libs/libopenblasp-r0-ae94cfde.3.9.dev.so and import numpy works without manual moves.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python, rust
Domain
backend, build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.