indygreg / indygreg/PyOxidizer

[help/bug?] dynamic importing of python modules from the file system

Open
#226 5 comments 0 reactions 0 assignees View on GitHub
bug enhancement
Dominant language
Rust
Stars
6.2k
Forks
256
PR merge metrics
No merged PRs in 30d

Description

We have a test application running `pytest` that uses some pip packages and some rust code.
The test writers develop mainly on the pytest tests, but rarely change the rust code and python packages. To keep iteration time low, I'm bundling the python interpreter, pip packages and rust code using pyoxidizer, but allowing users to run their pytest tests from the file system. Our build system scans the file system for changes, and then determines whether a rebuild of the executable (rust or pip changes), or just a re-run of the tests (pytest code changes) is necessary.

Pytest internally uses the (py)[https://github.com/pytest-dev/py] pacakge to dynamically import the test modules from the file system. It first finds the test file, adds the directory to the path, then uses `__import__` to load the module. (see https://github.com/pytest-dev/py/blob/master/py/_path/local.py#L701)

This breaks when using pyoxidizer, as it can't find the module, even if sys.path is correct. I've tried rewriting it to use `importlib.load_module` but that also fails.

I have locally modified copy of `py` that uses the code below instead of `__import__`, but I'm wondering if pyoxidier is intending to disallow the dynamic import of modules from the file system.

```python
spec = importlib.util.spec_from_file_location(modname, self.realpath())
new_mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(new_mod)
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the filesystem import failure with the pytest scenario described, then inspect py/_path/local.py around line 701 and PyOxidizer's handling of bundled Python modules. Compare the __import__ path with the importlib.util approach shown in the issue. Done means establishing whether this import pattern is supported and identifying a documented or tested resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
build-system, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.