indygreg / indygreg/PyOxidizer
OxidizedFinder not loading file based resource data from folders that aren't Python packages
- Dominant language
- Rust
- Stars
- 6.2k
- Forks
- 256
- PR merge metrics
- No merged PRs in 30d
Description
I'm relatively new to PyOxidizer, but I've been working for several days now to get one of my command line tools to run properly under the frozen environment and have hit a problem that looks to be a bug with PyOxidizer. Basically, the OxidizedFinder object does not seem to be loading data from resource files contained within Python packages properly. I've reduced the steps to reproduce the problem down to the following Starlark file:
```
def make_exe():
dist = default_python_distribution()
policy = dist.make_python_packaging_policy()
policy.resources_location_fallback = None
policy.resources_location = "filesystem-relative:lib"
python_config = dist.make_python_interpreter_config()
python_config.run_command = "import pkgutil; pkgutil.get_data('certifi', 'cacert.pem')"
exe = dist.to_python_executable(
name="pyapp",
packaging_policy=policy,
config=python_config,
)
for resource in exe.pip_install(["certifi"]):
exe.add_python_resource(resource)
return exe
def make_embedded_resources(exe):
return exe.to_embedded_resources()
def make_install(exe):
files = FileManifest()
files.add_python_resource(".", exe)
return files
def make_msi(exe):
return exe.to_wix_msi_builder(
"myapp",
"My Application",
"1.0",
"Alice Jones"
)
def register_code_signers():
if not VARS.get("ENABLE_CODE_SIGNING"):
return
register_code_signers()
register_target("exe", make_exe)
register_target("resources", make_embedded_resources, depends=["exe"], default_build_script=True)
register_target("install", make_install, depends=["exe"], default=True)
register_target("msi_installer", make_msi, depends=["exe"])
resolve_targets()
```
If you run "pyoxidizer run" against this build script it will fail with the following error:
```
Traceback (most recent call last):
File "", line 1, in
File "pkgutil", line 639, in get_data
FileNotFoundError: [Errno 2] resource not known: '/Users/kevinp/Documents/src/sandbox/oxydizer1/build/x86_64-apple-darwin/debug/install/lib/certifi/cacert.pem'
```
From what I can tell this error is generated by PyOxider from 1 of 2 places, [here](https://github.com/indygreg/PyOxidizer/blob/e31eabef1510dca1ec43fa4ed3d78ee27bb36b4d/pyembed/src/python_resources.rs#L974) or [here](https://github.com/indygreg/PyOxidizer/blob/e31eabef1510dca1ec43fa4ed3d78ee27bb36b4d/pyembed/src/python_resources.rs#L1062).
I will point out that this is a contrived example I created to help illustrate the problem I'm finding. The "certifi" library itself actually loads the indicated resource using [more robust methodologies](https://github.com/certifi/python-certifi/blob/109a6d7566a95229b10edf8300f354190faae9db/certifi/core.py) that work with PyOxidizer, but I've found quite a few other libraries that suffer from the same fundamental problem. For example, if you change the following lines in the Starlark file you'll get a similar error by simply importing the `jsonschema` library because it (for better or worse) [makes use](https://github.com/Julian/jsonschema/blob/836db7c4c7d9cede429a451201c1a4e059479f53/jsonschema/_utils.py#L52) of a pkgutil call in the [global namespace](https://github.com/Julian/jsonschema/blob/836db7c4c7d9cede429a451201c1a4e059479f53/jsonschema/validators.py#L352) which then fails immediately preventing you from even importing the library:
```
python_config.run_command = "from jsonschema import validate"
exe = dist.to_python_executable(
name="pyapp",
packaging_policy=policy,
config=python_config,
)
for resource in exe.pip_install(["jsonschema"]):
exe.add_python_resource(resource)
```
From what I can tell, any library that tries to use the OxidizedFinder, directly or indirectly, to load resource data simply fail outright with this error, even though I've confirmed the resource and path do exist and are correct.
*Analysis*
The definition for the `get_data` method exposed by the `pkgutil` module in the standard library can be found [here](https://github.com/python/cpython/blob/db3ff76da19004f266b62e98a81bdfd322861436/Lib/pkgutil.py#L599). If we look at that implementation and try running each operation in a REPL environment in pyoxidizer you can see some interesting things.
First I commented out the `python_config.run_command` line in the Starlark file and ran the interactive runtime using `pyoxidizer run`. In this environment I performed the following operations:
```
>>> package = "certifi"
>>> resource = "cacert.pem"
>>> import importlib
>>> import importlib.util
>>> spec = importlib.util.find_spec(package)
>>> spec
ModuleSpec(name='certifi', loader=, origin='/Users/kevinp/Documents/src/sandbox/oxydizer1/build/x86_64-apple-darwin/debug/install/lib/certifi/__init__.py', submodule_search_locations=['/Users/kevinp/Documents/src/sandbox/oxydizer1/build/x86_64-apple-darwin/debug/install/lib/certifi'])
>>> loader = spec.loader
>>> loader
>>> mod = importlib._bootstrap._load(spec)
>>> mod
>>> mod.__file__
'/Users/kevinp/Documents/src/sandbox/oxydizer1/build/x86_64-apple-darwin/debug/install/lib/certifi/__init__.py'
>>> import os
>>> resource_name = os.path.join(os.path.dirname(mod.__file__), resource)
>>> resource_name
'/Users/kevinp/Documents/src/sandbox/oxydizer1/build/x86_64-apple-darwin/debug/install/lib/certifi/cacert.pem'
>>> os.path.exists(resource_name)
True
>>> loader.get_data(resource_name)
Traceback (most recent call last):
File "", line 1, in
FileNotFoundError: [Errno 2] resource not known: '/Users/kevinp/Documents/src/sandbox/oxydizer1/build/x86_64-apple-darwin/debug/install/lib/certifi/cacert.pem'
```
As you can see from this output, the loader and module are created successfully, and the path to the data file is correct ... and it exists on disk, and yet the loader complains that the resource is unknown.
After doing some digging I found [this issue](https://github.com/indygreg/PyOxidizer/issues/139) that was created a couple of years ago, which is where the implementation for the `OxidizerFinder.get_data` method was implemented, and based on [this very detailed comment](https://github.com/indygreg/PyOxidizer/blob/e31eabef1510dca1ec43fa4ed3d78ee27bb36b4d/pyembed/src/python_resources.rs#L928) it appears as though there are additional requirements that resource files must satisfy before they will be loaded by PyOxidizer. Most notably, the files have be under the same path the application executable is stored. So I thought maybe I'm running into a path problem here. I'm somewhat familiar with Rust code so I tried reading over the implementation to see how those checks were being done, and tried reflecting my understanding into some Python equivalent operations to see I could find any further problems. Below is what I found:
```
>>> loader.origin
'/Users/kevinp/Documents/src/sandbox/oxydizer1/build/x86_64-apple-darwin/debug/install'
>>> resource_name.startswith(loader.origin)
True
>>> import io
>>> fh = io.FileIO(resource_name, "r")
>>> fh.read()
... dump of valid PEM file contents - too long to paste here...
```
So, based on these findings it would appear as though the logic from [this conditional block](https://github.com/indygreg/PyOxidizer/blob/e31eabef1510dca1ec43fa4ed3d78ee27bb36b4d/pyembed/src/python_resources.rs#L966) should be satisfied, meaning that [this line](https://github.com/indygreg/PyOxidizer/blob/e31eabef1510dca1ec43fa4ed3d78ee27bb36b4d/pyembed/src/python_resources.rs#L972) should not be the source of the error. This means that in all likelihood there is a bug later in this function which results in the resource file being incorrectly excluded from the loading logic, causing [this error](https://github.com/indygreg/PyOxidizer/blob/e31eabef1510dca1ec43fa4ed3d78ee27bb36b4d/pyembed/src/python_resources.rs#L1060) to be thrown.
Unfortunately, this is the extent of my expertise. I don't know enough about Rust to debug this issue further. Any assistance you can provide would be greatly appreciated.
*Environment
Host OS: Mac OS-X
Python version: 3.9.6
PyOxidizer version: 0.17.0
tested against latest versions of certifi and jsonschema
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with pyembed/src/python_resources.rs, especially OxidizedFinder.get_data and the referenced conditional and error paths. Reproduce the failure with the provided Starlark file and pyoxidizer run, then compare pkgutil.get_data behavior for certifi/cacert.pem and the jsonschema import. Done means filesystem-relative resource data loads successfully through OxidizedFinder without the reported FileNotFoundError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- build-system, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 42/100