indygreg / indygreg/PyOxidizer
Caching built C extensions
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 6.2k
- Forks
- 256
- PR merge metrics
- No merged PRs in 30d
Description
We are trying to get a single executable that has all the extension modules that we need as "build-in", and we found that running `pip_install` with flags to make it not use wheels for the packages that have C extensions works well:
```
exe.add_python_resources(exe.pip_install(["-r", "requirements.txt", "--no-binary", "psutil,grpcio"]))
```
The thing is - it takes a very long time to compile (specifically - `grpcio` is taking ~5m). We usually only change our own code (added with a `read_package_root` line after the line above), so it would be great if we could somehow not do the compilation again unless the requirements file changes (or we do some force cache cleaning).
To speed this up I can think of two options:
1. Caching the results of compilation on pip level - currently all the `pip install` have a `tempdir` as the target, but even changing it to a "constant" directory doesn't really fix anything because `pip install` still does the compilation again, even when the target directory already has it compile, I think because of https://github.com/pypa/pip/issues/9769, but the resolution for that seems to be stalled.
2. Building "in steps" - if we could have 2 different invocations of `pyoxidizer build` - one that would do everything up to and including installing the requirements, and then a separate invocation to just add our own sources and then compile the final app, that would solve everything for us, because we use a `Makefile` to run the build anyway, and can just rerun the first command only if `requirements.txt` changed, and run only the second phase if not.
Is there anything I can do in the `pyoxidizer.bzl` file to store `PythonExecutable` object in one target and then load it in a different target?
Any other ways to improve this?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the pyoxidizer.bzl example around pip_install, PythonExecutable, and read_package_root, then trace how requirements.txt is handled across repeated builds. Done means compiled C-extension requirements can be reused until requirements.txt changes or the cache is explicitly cleared, while changes to the application sources avoid recompilation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100