linkedin / linkedin/shiv

Slow run when large pyz file

Open
#208 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
114
PR merge metrics
No merged PRs in 30d

Description

I have "shived" all my dependencies (pandas, scipy, ...) to end up with a compressed .pyz file of close to 200MB.
My entry point is a function that does not import anything but modules from the sdlib (sys, pathlib) and runs immediately when run outside the shiv file.
However, when I run the shiv file, it takes 16s to run (once it has already been decompressed in the .shiv folder, for the first run, it takes 82s to decompress to ~600 MB and run)
Is there some way to improve this performance ?
Is the .pyz file completely read from disk & uncompressed each time the pyz file is run (even if it is only to check it has already been decompressed in the .shiv folder) ?
If so, would it make sense to split the file in two file, a small .pyz with the bootstrap logic (check .shiv/... folder is updated, run preamble,...) and a large file with the zipped site-packages that would only be accessed if there is a need to decompress for the first time to the .shiv folder ?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no repository files or tests; start by tracing the shiv bootstrap path, including the .pyz read, the .shiv directory check, and first-run decompression. Compare subsequent execution with initial extraction, and define done as avoiding unnecessary work on later runs without breaking dependency extraction.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.