indygreg / indygreg/PyOxidizer
Can't load shared libraries from Virtualenv built binaries
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 6.2k
- Forks
- 256
- PR merge metrics
- No merged PRs in 30d
Description
I have the following in my `pyoxidizer.bzl` file:
```
# Package things from a populated virtualenv.
packaging_rules.append(Virtualenv(path="../build/blah/venv"))
```
where `blah` is a project built using custom tooling. Lots of the dependencies of `blah` include shared libraries (e.g. `_psutil_osx`). But because the venv packages were *not* built by PyOxidizer, the `distutils` hack was never invoked and therefore we have no static libraries.
The resulting binary will exit on the `ImportError` but it might be possible to do *something* to support these, even if it's not as efficient as statically linking.
In another project, we wrote a small finder that can do this. The details probably won't be exactly the same, but it will give you an idea of what can be done:
```
class SOFinder:
def __init__(self, so_map, cache_root, archive):
self.so_map = so_map
self.cache_root = cache_root
self.archive = archive
def find_spec(self, name, paths, module=None):
origin = self.so_map.get(name)
if origin is None:
return None
extracted_path = Path(self.cache_root) / origin
if not extracted_path.exists():
self.archive.extract(origin, self.cache_root)
loader = ExtensionFileLoader(name, extracted_path)
return ModuleSpec(name, loader, origin=extracted_path)
```
Where `so_map` was a dict of so names to paths within a zip file.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Virtualenv(path="../build/blah/venv") packaging rule in pyoxidizer.bzl and trace how shared-library extensions are handled when the virtualenv was built externally. Compare that flow with the provided SOFinder example. Done means binaries built from such virtualenvs can import extensions such as _psutil_osx without requiring statically linked libraries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system, tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100