indygreg / indygreg/PyOxidizer
pkgutil.get_data() with in-memory resources
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 6.2k
- Forks
- 256
- PR merge metrics
- No merged PRs in 30d
Description
Third-party libraries like jsonschema, that call pkgutil.get_data() to read their bundled resources, currently fail when the sources are in memory, as they lack a `__file__` attribute. PyOxidizer's ResourceReader interface does not require `__file__`, and can successfully retrieve the resources. By dynamically replacing get_data() with a call to open_resource(), it appears to be possible to get jsonschema to work when in memory (mostly - it still has a dependency on an extension module):
```python
def make_exe():
dist = default_python_distribution()
policy = dist.make_python_packaging_policy()
policy.resources_location = "in-memory"
policy.resources_location_fallback = "filesystem-relative:lib"
python_config = dist.make_python_interpreter_config()
python_config.run_command = """
import sys
import pkgutil
import importlib
def get_data_custom(package, resource):
try:
module = importlib.import_module(package)
with module.__loader__.get_resource_reader(package).open_resource(resource) as f:
return f.read()
except:
return None
if getattr(sys, "oxidized", False):
pkgutil.get_data = get_data_custom
# quick jsonschema test
schema = {"type" : "object", "properties" : {"price" : {"type" : "number"}} }
import jsonschema
jsonschema.validate(instance=dict(price="test"), schema=schema)
"""
exe = dist.to_python_executable(
name = "pyapp",
packaging_policy = policy,
config = python_config,
)
for resource in exe.pip_install(["jsonschema"]):
exe.add_python_resource(resource)
return exe
def make_embedded_resources(exe):
return exe.to_embedded_resources()
def make_install(exe):
files = FileManifest()
files.add_python_resource(".", exe)
return files
register_target("exe", make_exe)
register_target("resources", make_embedded_resources, depends = ["exe"], default_build_script = True)
register_target("install", make_install, depends = ["exe"], default = True)
resolve_targets()
```
The sys.oxidized gate is necessary to ensure things run correctly outside PyOxidizer, as the default Python ResourceReader can not handle subdirs in the resource path.
I have not tested this with any other libraries yet, but presume this approach is not limited to jsonschema. It's a hack, and I'm not suggesting this be included in PyOxidizer itself, but I thought it might be worth mentioning in case anyone finds it useful.
Partially related to #436
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the supplied PyOxidizer configuration with jsonschema and compare pkgutil.get_data() against the ResourceReader open_resource() path for in-memory resources. Review the packaging policy and Python interpreter integration to determine whether standard resource loading should work without the proposed monkey patch; done would include a confirmed behavior and a regression test for the jsonschema case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100