indygreg / indygreg/PyOxidizer

Executable cannot start without PythonPackagingPolicy.include_distribution_sources = True

Open
#312 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
6.2k
Forks
256
PR merge metrics
No merged PRs in 30d

Description

I'm using PyOxidizer v0.9.0 on macOS.

# Reproducing the actual bug

Place the following in `pyoxidizer.bzl` in an otherwise empty directory and execute `pyoxidizer run`.

```bzl
def make():
dist = default_python_distribution()
python_config = dist.make_python_interpreter_config()
python_config.config_profile = "python"
python_config.run_mode = "eval:import sys; from pprint import pprint; pprint(sys.argv)"

policy = dist.make_python_packaging_policy()
policy.set_resource_handling_mode("classify")
policy.include_distribution_sources = False
policy.bytecode_optimize_level_zero = True
policy.include_distribution_resources = True
exe = dist.to_python_executable(
name="pyapp",
packaging_policy=policy,
config=python_config,
)
files = FileManifest()
files.add_python_resource(".", exe)
return files
register_target("install", make, default=True)
resolve_targets()
```

It compiles just fine. When it runs, it produces the following error message (paths shortened for clarity; also, despite the output below, my current shell session defines no `PYTHON*` environment variables):

```
Python path configuration:
PYTHONHOME = './build/x86_64-apple-darwin/debug/install'
PYTHONPATH = (not set)
program name = './build/x86_64-apple-darwin/debug/install/./pyapp'
isolated = 0
environment = 1
user site = 1
import site = 1
sys._base_executable = './build/x86_64-apple-darwin/debug/install/./pyapp'
sys.base_prefix = './build/x86_64-apple-darwin/debug/install'
sys.base_exec_prefix = './build/x86_64-apple-darwin/debug/install'
sys.executable = './build/x86_64-apple-darwin/debug/install/./pyapp'
sys.prefix = './build/x86_64-apple-darwin/debug/install'
sys.exec_prefix = './build/x86_64-apple-darwin/debug/install'
sys.path = [
'./build/x86_64-apple-darwin/debug/install/lib/python38.zip',
'./build/x86_64-apple-darwin/debug/install/lib/python3.8',
'./build/x86_64-apple-darwin/debug/install/lib/python3.8/lib-dynload',
]
during initializing Python main: init_fs_encoding: failed to get the Python codec of the filesystem encoding
```

My best guess is that the executable could not import the [`encodings`](https://github.com/python/cpython/tree/3.8/Lib/encodings) Python package:
[`init_fs_encoding`](https://github.com/python/cpython/blob/a12f459ec2a31b96a21c735eb18f3d0fd19e99ff/Objects/unicodeobject.c#L15787-L15806) calls [`config_get_codec_name`](https://github.com/python/cpython/blob/a12f459ec2a31b96a21c735eb18f3d0fd19e99ff/Objects/unicodeobject.c#L15680-L15724) calls [`_PyCodec_Lookup`](https://github.com/python/cpython/blob/a12f459ec2a31b96a21c735eb18f3d0fd19e99ff/Python/codecs.c#L85-L191) calls [`_PyCodecRegistry_Init`](https://github.com/python/cpython/blob/a12f459ec2a31b96a21c735eb18f3d0fd19e99ff/Python/codecs.c#L1415-L1539) [imports](https://github.com/python/cpython/blob/a12f459ec2a31b96a21c735eb18f3d0fd19e99ff/Python/codecs.c#L1532) `encodings`.

# Variations

The following modifications don't seem to fix the bug.
- Changing the `set_resource_handling_mode` mode to `"files"`
- Setting `bytecode_optimize_level_zero` to `False`
- Setting `include_distribution_resources` to `False`

# Expected Behavior

I would have expected
```bzl
policy.include_distribution_sources = False
policy.bytecode_optimize_level_zero = True
```
to include the Python distribution pure-Python code as byte code only without the source code. I had hoped this was the case because that would cut down on the size of the executable.

Once that failed, I was hoping that
```bzl
policy.include_distribution_sources = False
policy.include_distribution_resources = True
```
would force the inclusion of Python distribution resources where _resources_ refers to PyOxidizer Starlark types (as I explain below, I gather that this is not the case).

## Why I came up with this expectation

The [documentation](https://pyoxidizer.readthedocs.io/en/v0.9.0/config_type_python_packaging_policy.html) for `include_distribution_sources` and `include_distribution_resources` is a little vague. I'm going to quote it and some of its context at length because it all affects my understanding.

> **`include_distribution_sources`**
>
> (`bool`)
>
> Whether to add source code for Python modules in the Python distribution.
>
> Default is `True`.
>
> **`include_distribution_resources`**
>
> (`bool`)
>
> Whether to add Python package resources for Python packages in the Python distribution.
>
> Default is `False`.
>
> **`include_file_resources`**
>
> (`bool`)
>
> Whether [File](https://pyoxidizer.readthedocs.io/en/v0.9.0/config_type_file.html) resources have their `add_include` attribute set to `True` by default.
>
> Default is `False`.
>
> **`include_non_distribution_sources`**
>
> (`bool`)
>
> Whether to add source code for Python modules not in the Python distribution.

I understood `include_distribution_sources` and `include_non_distribution_sources` to have the same behavior but affecting `PythonModuleSource`s from the `PythonDistribution` versus, say, `PythonExecutable.pip_install`: If `bytecode_optimize_level_zero = True`, then the byte code is compiled and included in the binary regardless of whether the source code is also embedded in the binary. Setting `include_[non_]distribution_sources` to `False` doesn't preclude the importation of the relevant modules, but does exclude the source code from inclusion in the executable.

I understood `include_file_resources` and `include_distribution_resources` as being quite different from each other despite the overlapping use of the term _resource_. The resources that `include_file_resources` refers to are the PyOxidizer [`File`](https://pyoxidizer.readthedocs.io/en/v0.9.0/config_type_file.html) resource objects, whereas the resources that `include_distribution_resources` refers to are those readable at runtime with [`importlib.resources`](https://docs.python.org/3.8/library/importlib.html#module-importlib.resources).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the minimal pyoxidizer.bzl reproduction and run it with pyoxidizer run. Trace how PythonPackagingPolicy handles include_distribution_sources, bytecode_optimize_level_zero, and distribution resources, then verify the resulting executable starts and imports its filesystem encoding. Done means the documented policy combination produces a runnable executable without distribution source files.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.