python-poetry / python-poetry/poetry

Poetry fails to install project with FileExistsError when several instances of poetry are started simultaneously

Open
#4,436 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

kind/bug status/triage
Dominant language
Python
Stars
34.3k
Forks
2.5k
Avg merge
2d 19h
Merged PRs (30d)
30

Description

  • I am on the latest Poetry version.
  • I have searched the issues of this repo and believe that this is not a duplicate.
  • If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

Issue

Steps:

  1. Prepare pyproject.toml with some "heavy" packages in dependencies, e.g. https://gist.github.com/Phobosmir/85135e82b1b9a7c455c8b646c43e0954
  2. Copy it to different directories
  3. Run several instances of poetry install simultaneously

Expected:
All projects are successfully installed

Actual:
Some of installations fails with FileExistsError.
Reproduction rate is very high.

<...>
Stack trace:

  6  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\urllib3\response.py:438 in _error_catcher
      436│         try:
      437│             try:
    → 438│                 yield
      439│
      440│             except SocketTimeout:

  5  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\urllib3\response.py:519 in read
      517│             else:
      518│                 cache_content = False
    → 519│                 data = self._fp.read(amt) if not fp_closed else b""
      520│                 if (
      521│                     amt != 0 and not data

  4  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cachecontrol\filewrapper.py:65 in read
       63│         self.__buf.write(data)
       64│         if self.__is_fp_closed():
    →  65│             self._close()
       66│
       67│         return data

  3  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cachecontrol\filewrapper.py:52 in _close
       50│     def _close(self):
       51│         if self.__callback:
    →  52│             self.__callback(self.__buf.getvalue())
       53│
       54│         # We assign this to None here, because otherwise we can get into

  2  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cachecontrol\controller.py:308 in cache_response
      306│         if self.cache_etags and "etag" in response_headers:
      307│             logger.debug("Caching due to etag")
    → 308│             self.cache.set(
      309│                 cache_url, self.serializer.dumps(request, response, body=body)
      310│             )

  1  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cachecontrol\caches\file_cache.py:128 in set
      126│         with self.lock_class(name) as lock:
      127│             # Write our actual file
    → 128│             with _secure_open_write(lock.path, self.filemode) as fh:
      129│                 fh.write(value)
      130│

  FileExistsError

  [Errno 17] File exists: 'C:\\Users\\User\\AppData\\Local\\pypoetry\\Cache\\cache\\repositories\\pypi\\_http\\d\\a\\b\\8\\5\\dab85a2ce42d218d1f319dd5cfbe3d50b166447530e60aee10277972'

<...>

Ready batch scripts for reproduction could be found in repo:
https://github.com/Phobosmir/poetry-install-failure

When installing with pip + requirements.txt no issues could be found.

Tested on clean win10, no antiviruses or other staff. only git + python were installed.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with the batch scripts in the linked poetry-install-failure repository while running several installs against the same cache. Start at the cache write path shown in cachecontrol\caches\file_cache.py and investigate the FileExistsError on Windows. Done means simultaneous Poetry installs complete successfully without cache-related failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.