python-poetry / python-poetry/poetry
poetry install hugely slowed down by deepcopy operations
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 34.3k
- Forks
- 2.5k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 30
Description
- Poetry version: 1.3.2
- Python version: 3.9.11
- OS version and name: MacOS 12.5.1
- pyproject.toml: not shared
- I am on the latest stable Poetry version, installed using a recommended method.
- I have searched the issues of this repo and believe that this is not a duplicate.
- I have consulted the FAQ and blog for any relevant entries or release notes.
- If an exception occurs when executing a command, I executed it again in debug mode (
-vvvoption) and have included the output below.
Issue
poetry install takes considerable time (70 seconds) to run what should be a no-op. In fact:
- All packages to be installed are already cached
- All packages are already installed in the virtual environment. The command terminates with
No dependencies to install or update
Although I cannot share the pyproject.toml and poetry.lock files as they contain private dependencies (so you wouldn't be able to reproduce it), I can share a profiling of poetry that obtained running cProfile (link here). I'll share a few links that obtained plotting with snakeviz the result of
/path/to/pypoetry/venv/bin/python -m cProfile -o test.profile -m poetry install
As you can see lots of time seems to be spent in deepcopy operations in poetry-core.
I'll try to create a fully reproducible use case. In the meantime I can also share that:
- The project contains a great number of dependencies (around 200), some of them related to ML (PyTorch, TensorFlow, ..).
- The same issue happens in our CI systems running on Linux
- I experienced the same issue on both Poetry 1.2.2 ad Poetry 1.3.2
- While "stuck", the poetry process runs at full 100% CPU utilization
- We fetch all packages (including public ones) through Artifactory, a private packages repo. AFAIK Artifactory does not offer any JSON API like PyPI, which I know Poetry uses extensively
Thanks a lot for your support and for maintaining this amazing tool ❤️
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Begin with the supplied cProfile test.profile and the poetry install -vvv execution, then trace the reported poetry-core deepcopy calls. A useful fix would reproduce the slowdown with a representative large dependency set and make the no-op install complete without excessive deepcopy CPU time; the report provides no pyproject.toml or test target.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100