python-poetry / python-poetry/poetry
Poetry fails to install project with FileExistsError when several instances of poetry are started simultaneously
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 34.3k
- Forks
- 2.5k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 30
Description
- I am on the latest Poetry version.
- I have searched the issues of this repo and believe that this is not a duplicate.
- If an exception occurs when executing a command, I executed it again in debug mode (
-vvvoption).
- OS version and name: Windows 10 21H1 (Hyper-V default dev environment at a moment)
- Poetry version: 1.2.0a2 (c967a4a5); 1.1.7
- Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/Phobosmir/85135e82b1b9a7c455c8b646c43e0954
Issue
Steps:
- Prepare pyproject.toml with some "heavy" packages in dependencies, e.g. https://gist.github.com/Phobosmir/85135e82b1b9a7c455c8b646c43e0954
- Copy it to different directories
- Run several instances of poetry install simultaneously
Expected:
All projects are successfully installed
Actual:
Some of installations fails with FileExistsError.
Reproduction rate is very high.
<...>
Stack trace:
6 ~\AppData\Roaming\pypoetry\venv\lib\site-packages\urllib3\response.py:438 in _error_catcher
436│ try:
437│ try:
→ 438│ yield
439│
440│ except SocketTimeout:
5 ~\AppData\Roaming\pypoetry\venv\lib\site-packages\urllib3\response.py:519 in read
517│ else:
518│ cache_content = False
→ 519│ data = self._fp.read(amt) if not fp_closed else b""
520│ if (
521│ amt != 0 and not data
4 ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cachecontrol\filewrapper.py:65 in read
63│ self.__buf.write(data)
64│ if self.__is_fp_closed():
→ 65│ self._close()
66│
67│ return data
3 ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cachecontrol\filewrapper.py:52 in _close
50│ def _close(self):
51│ if self.__callback:
→ 52│ self.__callback(self.__buf.getvalue())
53│
54│ # We assign this to None here, because otherwise we can get into
2 ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cachecontrol\controller.py:308 in cache_response
306│ if self.cache_etags and "etag" in response_headers:
307│ logger.debug("Caching due to etag")
→ 308│ self.cache.set(
309│ cache_url, self.serializer.dumps(request, response, body=body)
310│ )
1 ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cachecontrol\caches\file_cache.py:128 in set
126│ with self.lock_class(name) as lock:
127│ # Write our actual file
→ 128│ with _secure_open_write(lock.path, self.filemode) as fh:
129│ fh.write(value)
130│
FileExistsError
[Errno 17] File exists: 'C:\\Users\\User\\AppData\\Local\\pypoetry\\Cache\\cache\\repositories\\pypi\\_http\\d\\a\\b\\8\\5\\dab85a2ce42d218d1f319dd5cfbe3d50b166447530e60aee10277972'
<...>
Ready batch scripts for reproduction could be found in repo:
https://github.com/Phobosmir/poetry-install-failure
When installing with pip + requirements.txt no issues could be found.
Tested on clean win10, no antiviruses or other staff. only git + python were installed.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with the batch scripts in the linked poetry-install-failure repository while running several installs against the same cache. Start at the cache write path shown in cachecontrol\caches\file_cache.py and investigate the FileExistsError on Windows. Done means simultaneous Poetry installs complete successfully without cache-related failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100