indygreg / indygreg/PyOxidizer

Feature Request: Dynamically linked bundle of classified resources

Open
#418 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
6.2k
Forks
256
PR merge metrics
No merged PRs in 30d

Description

In a typical Python scientific computing project, the on-disk size is probably dominated by large dependencies like `scipy` and `numpy` (88 and 46 MB in my build, respectively), not to mention the standard library. These dependencies likely change at a far slower rate than the local Python and Rust code that makes up the rest of the project. Having to redistribute a 100 MB+ bundle every update is less than ideal.

Granted, those large dependencies are also `__file__` users, and must be stored in a "lib" folder anyway. For now, it's possible to leave those alone and distribute the smaller binary (39 MB in my build) instead. However, that binary still contains the stdlib and other packages that can be classified, because I'd like to make use of the fast, zero-copy imports where I can. In future, once the package resources problem is (more) resolved, (more of) those dependencies will make it into the binary to enjoy the same benefits. This is key, I think: the benefit from packing dependencies into the binary is not _just_ "single file" distribution.

It would be really neat if we could package rarely-changing dependencies into a "dependencies.dll" file, or similar, to preserve fast imports while reducing update size. Dynamic linking being what it is, I imagine that this would be very nontrivial to implement cross-platform. Or maybe not; this is outside my area of expertise. Just wanted to raise it as a suggestion!

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are named in the issue. Start by determining how PyOxidizer currently classifies and packages Python and Rust resources; done would mean a cross-platform design or implementation for dynamically linked dependency bundles that preserves fast imports while reducing update sizes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
build-system
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.