indygreg / indygreg/PyOxidizer

`bytecode_optimize_level_* = True` incompatible with `add_location = 'filesystem-relative:*'`

Open
#417 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
6.2k
Forks
256
PR merge metrics
No merged PRs in 30d

Description

Windows 10, PyOxidizer 0.17.0-pre

I'm building a project that depends on `numpy`, `matplotlib`, `scipy` and friends (all of which use `__file__` and have not-yet-automatically-detectable shared libraries), and I found https://github.com/indygreg/PyOxidizer/issues/303#issuecomment-715992218 to be very helpful.

I've got the build script to a point where everything's included nicely (only `__file__` packages on filesystem and using `File`, everything else using "Classified" and in-memory) and functioning correctly, so the next step was to use bytecode over module sources since the whole bundle is still 254.7 MB.

- Setting `policy.include_non_distribution_sources = False` and `policy.bytecode_optimize_level_two` globally doesn't work. The .py files are still on the filesystem, and the imports start failing with `__file__` again. Okay, this makes sense. I was emitting both classified and unclassified resources, always including the classified, including the unclassified only on the modules hitting the filesystem, setting ALL resources for the filesystem modules to "filesystem-relative:lib", and relying on the filesystem to dedupe. That worked before, but it looks like now the bytecode is going into the binary and leaving the `File` .py files on the filesystem. The in-memory bytecode takes import precedence and `__file__` fails again. Got it.
- Returning the `policy` source/bytecode settings to default and using the callback to set only non-filesystem `PythonModuleSource`s to "no source", "yes bytecode 2" works and results in a 243.6 MB bundle. Not bad! The filesystem's still using sources, though...
- Ok, so `File`s don't seem to respect bytecode setting. Makes sense, I guess, because they're unclassified. So, filter out the `File`s with `resource.path.endswith(".py")`, pretty straightforward. Set _all_ `PythonModuleSource`s to "no source", "yes bytecode 2". Now the bundle's 233.7 MB, the .py files are gone, no bytecode has showed up in their place, and `__file__` problems are back. `print` debugging confirms that the `PythonModuleSource`s _are_ set to "filesystem-relative:lib"

I'm reasonably concluding that the bytecode compilation machinery only works for the in-memory option, and that it overrides the `add_location` instead of failing. Haven't dug into the source yet to determine whether this is an oversight, a current known limitation, or a logic error, but I'm pretty sure that it's happening.

Contributor guide

No contributing guide indexed for this repository

Research direction

No source files or tests are named. Reproduce the reported Windows configuration with filesystem-relative add_location and bytecode optimization, then trace the bytecode compilation and resource-location handling in PyOxidizer. Done means the interaction is fixed or its limitation is explicitly documented with a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
build-system, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.