python-poetry / python-poetry/poetry

poetry install hugely slowed down by deepcopy operations

Open
#7,459 7 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

kind/bug status/triage
Dominant language
Python
Stars
34.3k
Forks
2.5k
Avg merge
2d 19h
Merged PRs (30d)
30

Description

  • Poetry version: 1.3.2
  • Python version: 3.9.11
  • OS version and name: MacOS 12.5.1
  • pyproject.toml: not shared
  • I am on the latest stable Poetry version, installed using a recommended method.
  • I have searched the issues of this repo and believe that this is not a duplicate.
  • I have consulted the FAQ and blog for any relevant entries or release notes.
  • If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

poetry install takes considerable time (70 seconds) to run what should be a no-op. In fact:

  • All packages to be installed are already cached
  • All packages are already installed in the virtual environment. The command terminates with No dependencies to install or update

Although I cannot share the pyproject.toml and poetry.lock files as they contain private dependencies (so you wouldn't be able to reproduce it), I can share a profiling of poetry that obtained running cProfile (link here). I'll share a few links that obtained plotting with snakeviz the result of

/path/to/pypoetry/venv/bin/python -m cProfile -o test.profile -m poetry install

As you can see lots of time seems to be spent in deepcopy operations in poetry-core.

I'll try to create a fully reproducible use case. In the meantime I can also share that:

  • The project contains a great number of dependencies (around 200), some of them related to ML (PyTorch, TensorFlow, ..).
  • The same issue happens in our CI systems running on Linux
  • I experienced the same issue on both Poetry 1.2.2 ad Poetry 1.3.2
  • While "stuck", the poetry process runs at full 100% CPU utilization
  • We fetch all packages (including public ones) through Artifactory, a private packages repo. AFAIK Artifactory does not offer any JSON API like PyPI, which I know Poetry uses extensively

Thanks a lot for your support and for maintaining this amazing tool ❤️

Screenshot 2023-02-02 at 15 40 25 Screenshot 2023-02-02 at 15 40 41 Screenshot 2023-02-02 at 15 41 00

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Begin with the supplied cProfile test.profile and the poetry install -vvv execution, then trace the reported poetry-core deepcopy calls. A useful fix would reproduce the slowdown with a representative large dependency set and make the no-op install complete without excessive deepcopy CPU time; the report provides no pyproject.toml or test target.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.