indygreg / indygreg/PyOxidizer

Including itertools from cpython

Open
#407 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
6.2k
Forks
256
PR merge metrics
No merged PRs in 30d

Description

Hi!
First of all, I want to say thank you for developing this tool. I am a big fan of Rust, and this tool is probably the easiest packaging manager I've used so far (if I can call it that).

The issue:
I'm still trying to figure out how to properly make a configuration file, so I may have just overlooked this in the documentation, but I can't seem to include itertools in the final build.
My script runs fine if I build with `policy.extension_module_filter = "all"`, but I would really rather use the `minimal` option and then fine-tune other included packages.
I've tried a couple of different things to see if I could make it work, but to no avail.

My configuration file is basically the following:
```python
def make_dist():
return default_python_distribution()

def resource_callback(policy, resource):
if resource.package == "itertools":
resource.add_include = True
# ^^^^ tried this

def make_exe(dist):
policy = dist.make_python_packaging_policy()
policy.allow_in_memory_shared_library_loading = True
policy.bytecode_optimize_level_two = True
policy.file_scanner_classify_files = True
policy.file_scanner_emit_files = False
policy.include_classified_resources = True
policy.extension_module_filter = "minimal"
policy.resources_location = "in-memory"
policy.resources_location_fallback = "filesystem-relative:lib"
policy.register_resource_callback(resource_callback)

python_config = dist.make_python_interpreter_config()
python_config.filesystem_importer = True
python_config.run_module = "my_script"

exe = dist.to_python_executable(
name="my_script",
packaging_policy=policy,
config=python_config,
)
exe.add_python_resources([
PythonExtensionModule(
name="itertools",
is_stdlib=True,
add_include=True,
add_location="in_memory",
add_location_fallback="filesystem-relative:lib",
add_bytecod_optimization_level_two=True
),
# ^^^^ also tried this

exe.read_package_root(
path="my_path",
packages=["my_script"]
),
exe.read_package_root(
path="my_path/src/py",
packages=["local_package1", "local_package2", "local_package3"]
),
exe.pip_install(["-r", "req.txt"]
)
])
return exe

def make_embedded_resources(exe):
return exe.to_embedded_resources()

def make_install(exe):
# Create an object that represents our installed application file layout.
files = FileManifest()

# Add the generated executable to our install layout in the root directory.
files.add_python_resource(".", exe)

return files

# Tell PyOxidizer about the build targets defined above.
register_target("dist", make_dist)
register_target("exe", make_exe, depends=["dist"])
register_target("resources", make_embedded_resources, depends=["exe"], default_build_script=True)
register_target("install", make_install, depends=["exe"], default=True)

# Resolve whatever targets the invoker of this configuration file is requesting
# be resolved.
resolve_targets()

# END OF COMMON USER-ADJUSTED SETTINGS.
#
# Everything below this is typically managed by PyOxidizer and doesn't need
# to be updated by people.

PYOXIDIZER_VERSION = "0.16.2"
PYOXIDIZER_COMMIT = "e91995636f8deed0a7d8e1917f96a7dc17309b63"
```

However, I'm guessing neither of the things I tried in the above code are not the right approach, as the callback results in:
```
error[PYOXIDIZER_BUILD]: error converting PythonResource to Value: DiagnosedError(Diagnostic { level: Error, message: "Cannot .package on type PythonExtensionModule", code: Some("CV00"), spans: [SpanLabel { span: Span { low: Pos(138), high: Pos(154) }, label: Some(".package not supported for type PythonExtensionModule"), style: Primary }] })
--> ./pyoxidizer.bzl:30:11
|
30 | exe = dist.to_python_executable(
| ___________^
31 | | name="my_script",
32 | | packaging_policy=policy,
33 | | config=python_config,
34 | | )
| |_____^ to_python_executable()

error: error converting PythonResource to Value: DiagnosedError(Diagnostic { level: Error, message: "Cannot .package on type PythonExtensionModule", code: Some("CV00"), spans: [SpanLabel { span: Span { low: Pos(138), high: Pos(154) }, label: Some(".package not supported for type PythonExtensionModule"), style: Primary }] })
```

and the attempt to directly create a PythonExtensionModule results in:
```
error[CM01]: Variable 'PythonExtensionModule' not found
--> ./pyoxidizer.bzl:31:9
|
31 | PythonExtensionModule(
| ^^^^^^^^^^^^^^^^^^^^^ Variable was not found

error: Variable 'PythonExtensionModule' not found
```

Sorry if this is just something I missed in the documentation, but can someone please help me understand the proper way to fine tune which packages are included in the build when using ` policy.extension_module_filter = "minimal"`?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the provided pyoxidizer.bzl make_exe entry point, including resource_callback and the attempted PythonExtensionModule resource, then inspect how extension_module_filter="minimal" handles standard-library modules. Verify the supported configuration path by building with minimal filtering and confirming that itertools is available in the resulting executable.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
build-system, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.