indygreg / indygreg/PyOxidizer

pkgutil.get_data() with in-memory resources

Open
#457 5 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug enhancement
Dominant language
Rust
Stars
6.2k
Forks
256
PR merge metrics
No merged PRs in 30d

Description

Third-party libraries like jsonschema, that call pkgutil.get_data() to read their bundled resources, currently fail when the sources are in memory, as they lack a `__file__` attribute. PyOxidizer's ResourceReader interface does not require `__file__`, and can successfully retrieve the resources. By dynamically replacing get_data() with a call to open_resource(), it appears to be possible to get jsonschema to work when in memory (mostly - it still has a dependency on an extension module):

```python
def make_exe():
dist = default_python_distribution()

policy = dist.make_python_packaging_policy()
policy.resources_location = "in-memory"
policy.resources_location_fallback = "filesystem-relative:lib"

python_config = dist.make_python_interpreter_config()
python_config.run_command = """
import sys
import pkgutil
import importlib

def get_data_custom(package, resource):
try:
module = importlib.import_module(package)
with module.__loader__.get_resource_reader(package).open_resource(resource) as f:
return f.read()
except:
return None
if getattr(sys, "oxidized", False):
pkgutil.get_data = get_data_custom

# quick jsonschema test
schema = {"type" : "object", "properties" : {"price" : {"type" : "number"}} }

import jsonschema
jsonschema.validate(instance=dict(price="test"), schema=schema)
"""

exe = dist.to_python_executable(
name = "pyapp",
packaging_policy = policy,
config = python_config,
)
for resource in exe.pip_install(["jsonschema"]):
exe.add_python_resource(resource)
return exe

def make_embedded_resources(exe):
return exe.to_embedded_resources()

def make_install(exe):
files = FileManifest()
files.add_python_resource(".", exe)
return files

register_target("exe", make_exe)
register_target("resources", make_embedded_resources, depends = ["exe"], default_build_script = True)
register_target("install", make_install, depends = ["exe"], default = True)

resolve_targets()

```

The sys.oxidized gate is necessary to ensure things run correctly outside PyOxidizer, as the default Python ResourceReader can not handle subdirs in the resource path.

I have not tested this with any other libraries yet, but presume this approach is not limited to jsonschema. It's a hack, and I'm not suggesting this be included in PyOxidizer itself, but I thought it might be worth mentioning in case anyone finds it useful.

Partially related to #436

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the supplied PyOxidizer configuration with jsonschema and compare pkgutil.get_data() against the ResourceReader open_resource() path for in-memory resources. Review the packaging policy and Python interpreter integration to determine whether standard resource loading should work without the proposed monkey patch; done would include a confirmed behavior and a regression test for the jsonschema case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.