indygreg / indygreg/PyOxidizer
`bytecode_optimize_level_* = True` incompatible with `add_location = 'filesystem-relative:*'`
- Dominant language
- Rust
- Stars
- 6.2k
- Forks
- 256
- PR merge metrics
- No merged PRs in 30d
Description
Windows 10, PyOxidizer 0.17.0-pre
I'm building a project that depends on `numpy`, `matplotlib`, `scipy` and friends (all of which use `__file__` and have not-yet-automatically-detectable shared libraries), and I found https://github.com/indygreg/PyOxidizer/issues/303#issuecomment-715992218 to be very helpful.
I've got the build script to a point where everything's included nicely (only `__file__` packages on filesystem and using `File`, everything else using "Classified" and in-memory) and functioning correctly, so the next step was to use bytecode over module sources since the whole bundle is still 254.7 MB.
- Setting `policy.include_non_distribution_sources = False` and `policy.bytecode_optimize_level_two` globally doesn't work. The .py files are still on the filesystem, and the imports start failing with `__file__` again. Okay, this makes sense. I was emitting both classified and unclassified resources, always including the classified, including the unclassified only on the modules hitting the filesystem, setting ALL resources for the filesystem modules to "filesystem-relative:lib", and relying on the filesystem to dedupe. That worked before, but it looks like now the bytecode is going into the binary and leaving the `File` .py files on the filesystem. The in-memory bytecode takes import precedence and `__file__` fails again. Got it.
- Returning the `policy` source/bytecode settings to default and using the callback to set only non-filesystem `PythonModuleSource`s to "no source", "yes bytecode 2" works and results in a 243.6 MB bundle. Not bad! The filesystem's still using sources, though...
- Ok, so `File`s don't seem to respect bytecode setting. Makes sense, I guess, because they're unclassified. So, filter out the `File`s with `resource.path.endswith(".py")`, pretty straightforward. Set _all_ `PythonModuleSource`s to "no source", "yes bytecode 2". Now the bundle's 233.7 MB, the .py files are gone, no bytecode has showed up in their place, and `__file__` problems are back. `print` debugging confirms that the `PythonModuleSource`s _are_ set to "filesystem-relative:lib"
I'm reasonably concluding that the bytecode compilation machinery only works for the in-memory option, and that it overrides the `add_location` instead of failing. Haven't dug into the source yet to determine whether this is an oversight, a current known limitation, or a logic error, but I'm pretty sure that it's happening.
Contributor guide
No contributing guide indexed for this repository
Research direction
No source files or tests are named. Reproduce the reported Windows configuration with filesystem-relative add_location and bytecode optimization, then trace the bytecode compilation and resource-location handling in PyOxidizer. Done means the interaction is fixed or its limitation is explicitly documented with a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- build-system, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100