No such file or directory due to race condition

• Poetry version: 1.5.1
• Python version: 3.10.4
• OS version and name: Ubuntu 22
• pyproject.toml:

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

This is similar to6958 but is not a duplicate as its a different use case.

This was happening for us on 1.3.1 and previous versions too.

Issue

Poetry fails to either install or upgrade a package (in this case setuptools) because it fails to find the .dist-info directory of another package (in this case proto_plus-1.22.2.dist-info) and that's because that package was upgraded from 1.22.2 to 1.22.3 just before (see output) but I suspect that operation didn't update the register in time.

In this case it's the pip uninstall which fails:

 ['/usr/local/bin/python3.10', '-m', 'pip', 'uninstall', 'setuptools', '-y']

(copied from output below)

But the same issue can happen with a pip install too:

'['/usr/local/bin/python3.10', '-m', 'pip', 'install', '--disable-pip-version-check', '--isolated', '--no-input', '--prefix', '/usr/local', '--no-deps', '/root/.cache/pypoetry/artifacts/2a/5a/46/20df50732755163cfab8d91b52f952990368c16a4c1005b865230dca24/google_cloud_core-2.3.2-py2.py3-none-any.whl']' returned non-zero exit status 1.

(taken from a different run that failed)

Re-running the job usually works, but it can fail on other packages or other projects in the job. This happens on maybe 1 out of 50-100 installs, but seeing we have 30+ projects/installs per job, jobs as a whole fail quite frequently. This has been happening on and off for well over a year, but weirdly it started happening with increasing frequency lately (on 1.3.1) and we just updated to 1.5.1 hoping it might resolve but to no avail.

There is no change to our CI environment that we are aware of (but it could even be a network change causing some packages to install much faster than before/on some occasions). We have no record of this happening locally but that could be down to number of installs.

I suspect this is related to the race condition fixed by 6186 and suspect that if we switch off parallel installations that fixes it.

The output below is from a private runner for github CI, and the command is running against a docker container.

Skipping virtualenv creation, as specified in config file.
docker run -v /runner-tmp/np-v2/np-v2/.venv/lib/python3.10/site-packages:/usr/local/lib/python3.10/site-packages  eu.gcr.io/np-root-project/np-db-utils:f2141c4a bash -c "PIP_ROOT_USER_ACTION=ignore poetry install"
Installing dependencies from lock file

Package operations: 1 install, 27 updates, 0 removals

  • Updating protobuf (4.23.2 -> 4.23.3)
  • Updating exceptiongroup (1.1.1 -> 1.1.2)
  • Updating google-auth (2.19.1 -> 2.21.0)
  • Updating googleapis-common-protos (1.59.0 -> 1.59.1)
  • Updating grpcio (1.54.2 -> 1.56.0)
  • Updating pyparsing (3.0.9 -> 3.1.0)
  • Updating google-api-core (2.11.0 -> 2.11.1)
  • Updating grpcio-status (1.54.2 -> 1.56.0)
  • Updating typing-extensions (4.6.3 -> 4.7.1)
  • Updating platformdirs (3.5.1 -> 3.8.0)
  • Updating proto-plus (1.22.2 -> 1.22.3)
  • Updating pydantic (1.10.8 -> 1.10.10)
  • Updating redis (4.5.5 -> 4.6.0)
  • Updating setuptools (67.8.0 -> 68.0.0)

  CalledProcessError

  Command '['/usr/local/bin/python3.10', '-m', 'pip', 'uninstall', 'setuptools', '-y']' returned non-zero exit status 2.

  at /usr/local/lib/python3.10/subprocess.py:524 in run
       520│             # We don't call process.wait() as .__exit__ does that for us.
       521│             raise
       522│         retcode = process.poll()
       523│         if check and retcode:
    →  524│             raise CalledProcessError(retcode, process.args,
       525│                                      output=stdout, stderr=stderr)
       526│     return CompletedProcess(process.args, retcode, stdout, stderr)
       527│ 
       528│ 

The following error occurred when trying to handle this error:


  EnvCommandError

  Command ['/usr/local/bin/python3.10', '-m', 'pip', 'uninstall', 'setuptools', '-y'] errored with the following return code 2
  
  Output:
  ERROR: Exception:
  Traceback (most recent call last):
    File "/usr/local/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
      status = run_func(*args)
    File "/usr/local/lib/python3.10/site-packages/pip/_internal/commands/uninstall.py", line 60, in run
      session = self.get_default_session(options)
    File "/usr/local/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 75, in get_default_session
      self._session = self.enter_context(self._build_session(options))
    File "/usr/local/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 89, in _build_session
      session = PipSession(
    File "/usr/local/lib/python3.10/site-packages/pip/_internal/network/session.py", line 282, in __init__
      self.headers["User-Agent"] = user_agent()
    File "/usr/local/lib/python3.10/site-packages/pip/_internal/network/session.py", line 157, in user_agent
      setuptools_dist = get_default_environment().get_distribution("setuptools")
    File "/usr/local/lib/python3.10/site-packages/pip/_internal/metadata/__init__.py", line 24, in get_default_environment
      from .pkg_resources import Environment
    File "/usr/local/lib/python3.10/site-packages/pip/_internal/metadata/pkg_resources.py", line 9, in <module>
      from pip._vendor import pkg_resources
    File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3252, in <module>
      def _initialize_master_working_set():
    File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pkg_resources/__init__.py", line 32[35](https://github.com/NetPurpose/np-v2/actions/runs/5465733537/jobs/9950196335#step:16:36), in _call_aside
      f(*args, **kwargs)
    File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3287, in _initialize_master_working_set
      list(map(working_set.add_entry, sys.path))
    File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pkg_resources/__init__.py", line 623, in add_entry
      for dist in find_distributions(entry, True):
    File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2065, in find_on_path
      for dist in factory(fullpath):
    File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2127, in distributions_from_metadata
      if len(os.listdir(path)) == 0:
  FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.10/site-packages/proto_plus-1.22.2.dist-info'
  

  at /opt/poetry/venv/lib/python3.10/site-packages/poetry/utils/env.py:1[52](https://github.com/NetPurpose/np-v2/actions/runs/5465733537/jobs/9950196335#step:16:53)5 in _run
      1521│                 output = ""
      1522│             else:
      1523│                 output = subprocess.check_output(cmd, stderr=stderr, env=env, **kwargs)
      1524│         except CalledProcessError as e:
    → 1525│             raise EnvCommandError(e, input=input_)
      1526│ 
      1527│         return decode(output)
      1528│ 
      1529│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:

Error: Process completed with exit code 1.

Start with poetry/utils/env.py around _run (line 1525), then compare the parallel-installation race fix in PR #6186 and reproduce the pip uninstall/install failure in the reported Docker and Ubuntu environment. Done means the missing .dist-info race no longer causes Poetry installs or upgrades to fail, with coverage for the reported sequence.