[Feature] Include `SHIV_COMPILE_WORKERS` in `environment.json`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 114
- PR merge metrics
- No merged PRs in 30d
Description
Problem
tl;dr --compile-pyc is a potential footgun that is not well articulated in the documentation. It can provide legitimate benefits but to be entirely safe it needs the ability to put into place additional guardrails.
I've seen a number of teams at various organizations use shiv to make easy to distribute internal tooling. Once again, however, I've seen a team that distributed a tool and that tool was used on an environment that had 94 logical CPUs assigned to it. This caused shiv bootstrap (via compileall) to spin up 94 child processes for pyc compilation. This then caused the tool to hang without any visibility as it was part of a cron job because the preamble happens after extraction and compilation and this hang happens before then.
This is related to #54 and #200 but I'd like to propose a different solution since we've already turned on --no-compile-pyc to avoid this going forward.
Potential Solution
Allow the build stage to include a cap on the number of compilation workers. This is possible at execution time with SHIV_COMPILE_WORKERS but not every downstream consumer of a zipapp created by shiv will know:
- That it needs to set this environment variable
- That this zipapp was created with shiv instead of pex or another tool
- That they're getting a Python zipapp executable
If this can be specified and stored in environment.json then this solves the problem of there being way more potential CPUs than there is work and a deadlock trying to compile the site-packages directory.
This also means that the person creating the zipapp can decide what might be a reasonable cap.
I would combine the storage of compile_workers in environment.json with a min(compile_workers, os.process_cpu_count()) so if the actual CPU count is lower than what was specified, we still use that instead of what was set.
Edit to add: I'm happy to send a PR if this seems reasonable.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the build stage that writes environment.json and the bootstrap path that invokes compileall; inspect how SHIV_COMPILE_WORKERS is currently read. Done means the worker cap is stored in environment.json, applied during compilation, and limited by the available CPU count. Run the existing test suite to verify the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system, cli
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100