indygreg / indygreg/PyOxidizer
Executable cannot start without PythonPackagingPolicy.include_distribution_sources = True
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 6.2k
- Forks
- 256
- PR merge metrics
- No merged PRs in 30d
Description
I'm using PyOxidizer v0.9.0 on macOS.
# Reproducing the actual bug
Place the following in `pyoxidizer.bzl` in an otherwise empty directory and execute `pyoxidizer run`.
```bzl
def make():
dist = default_python_distribution()
python_config = dist.make_python_interpreter_config()
python_config.config_profile = "python"
python_config.run_mode = "eval:import sys; from pprint import pprint; pprint(sys.argv)"
policy = dist.make_python_packaging_policy()
policy.set_resource_handling_mode("classify")
policy.include_distribution_sources = False
policy.bytecode_optimize_level_zero = True
policy.include_distribution_resources = True
exe = dist.to_python_executable(
name="pyapp",
packaging_policy=policy,
config=python_config,
)
files = FileManifest()
files.add_python_resource(".", exe)
return files
register_target("install", make, default=True)
resolve_targets()
```
It compiles just fine. When it runs, it produces the following error message (paths shortened for clarity; also, despite the output below, my current shell session defines no `PYTHON*` environment variables):
```
Python path configuration:
PYTHONHOME = './build/x86_64-apple-darwin/debug/install'
PYTHONPATH = (not set)
program name = './build/x86_64-apple-darwin/debug/install/./pyapp'
isolated = 0
environment = 1
user site = 1
import site = 1
sys._base_executable = './build/x86_64-apple-darwin/debug/install/./pyapp'
sys.base_prefix = './build/x86_64-apple-darwin/debug/install'
sys.base_exec_prefix = './build/x86_64-apple-darwin/debug/install'
sys.executable = './build/x86_64-apple-darwin/debug/install/./pyapp'
sys.prefix = './build/x86_64-apple-darwin/debug/install'
sys.exec_prefix = './build/x86_64-apple-darwin/debug/install'
sys.path = [
'./build/x86_64-apple-darwin/debug/install/lib/python38.zip',
'./build/x86_64-apple-darwin/debug/install/lib/python3.8',
'./build/x86_64-apple-darwin/debug/install/lib/python3.8/lib-dynload',
]
during initializing Python main: init_fs_encoding: failed to get the Python codec of the filesystem encoding
```
My best guess is that the executable could not import the [`encodings`](https://github.com/python/cpython/tree/3.8/Lib/encodings) Python package:
[`init_fs_encoding`](https://github.com/python/cpython/blob/a12f459ec2a31b96a21c735eb18f3d0fd19e99ff/Objects/unicodeobject.c#L15787-L15806) calls [`config_get_codec_name`](https://github.com/python/cpython/blob/a12f459ec2a31b96a21c735eb18f3d0fd19e99ff/Objects/unicodeobject.c#L15680-L15724) calls [`_PyCodec_Lookup`](https://github.com/python/cpython/blob/a12f459ec2a31b96a21c735eb18f3d0fd19e99ff/Python/codecs.c#L85-L191) calls [`_PyCodecRegistry_Init`](https://github.com/python/cpython/blob/a12f459ec2a31b96a21c735eb18f3d0fd19e99ff/Python/codecs.c#L1415-L1539) [imports](https://github.com/python/cpython/blob/a12f459ec2a31b96a21c735eb18f3d0fd19e99ff/Python/codecs.c#L1532) `encodings`.
# Variations
The following modifications don't seem to fix the bug.
- Changing the `set_resource_handling_mode` mode to `"files"`
- Setting `bytecode_optimize_level_zero` to `False`
- Setting `include_distribution_resources` to `False`
# Expected Behavior
I would have expected
```bzl
policy.include_distribution_sources = False
policy.bytecode_optimize_level_zero = True
```
to include the Python distribution pure-Python code as byte code only without the source code. I had hoped this was the case because that would cut down on the size of the executable.
Once that failed, I was hoping that
```bzl
policy.include_distribution_sources = False
policy.include_distribution_resources = True
```
would force the inclusion of Python distribution resources where _resources_ refers to PyOxidizer Starlark types (as I explain below, I gather that this is not the case).
## Why I came up with this expectation
The [documentation](https://pyoxidizer.readthedocs.io/en/v0.9.0/config_type_python_packaging_policy.html) for `include_distribution_sources` and `include_distribution_resources` is a little vague. I'm going to quote it and some of its context at length because it all affects my understanding.
> **`include_distribution_sources`**
>
> (`bool`)
>
> Whether to add source code for Python modules in the Python distribution.
>
> Default is `True`.
>
> **`include_distribution_resources`**
>
> (`bool`)
>
> Whether to add Python package resources for Python packages in the Python distribution.
>
> Default is `False`.
>
> **`include_file_resources`**
>
> (`bool`)
>
> Whether [File](https://pyoxidizer.readthedocs.io/en/v0.9.0/config_type_file.html) resources have their `add_include` attribute set to `True` by default.
>
> Default is `False`.
>
> **`include_non_distribution_sources`**
>
> (`bool`)
>
> Whether to add source code for Python modules not in the Python distribution.
I understood `include_distribution_sources` and `include_non_distribution_sources` to have the same behavior but affecting `PythonModuleSource`s from the `PythonDistribution` versus, say, `PythonExecutable.pip_install`: If `bytecode_optimize_level_zero = True`, then the byte code is compiled and included in the binary regardless of whether the source code is also embedded in the binary. Setting `include_[non_]distribution_sources` to `False` doesn't preclude the importation of the relevant modules, but does exclude the source code from inclusion in the executable.
I understood `include_file_resources` and `include_distribution_resources` as being quite different from each other despite the overlapping use of the term _resource_. The resources that `include_file_resources` refers to are the PyOxidizer [`File`](https://pyoxidizer.readthedocs.io/en/v0.9.0/config_type_file.html) resource objects, whereas the resources that `include_distribution_resources` refers to are those readable at runtime with [`importlib.resources`](https://docs.python.org/3.8/library/importlib.html#module-importlib.resources).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the minimal pyoxidizer.bzl reproduction and run it with pyoxidizer run. Trace how PythonPackagingPolicy handles include_distribution_sources, bytecode_optimize_level_zero, and distribution resources, then verify the resulting executable starts and imports its filesystem encoding. Done means the documented policy combination produces a runnable executable without distribution source files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- build-system
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100