[BUG] WorkingSet no longer prefers latest version of a dist

setuptools version

setuptools==66.0.0

Python version

Python 3.11

OS

Fedora 37

Additional environment information

No response

Description

Given that there are multiple versions of a distribution installed under a single path entry, calling WorkingSet.require with the latest version of that distribution results in a VersionConflict where it previously succeeded.

On my Fedora system, during WorkingSet._build_master, this consequently results in a call to WorkingSet._build_from_requirements where previously the call to WorkingSet.require succeeded, which in turn results in a new InvalidVersion exception which would otherwise not have occurred (unfortunately in an unrelated system package). I understand that the system package should be fixed, but this change in behavior exacerbates that issue where I would otherwise not have been impacted.

Here is the PR which modified the order of the distributions: https://github.com/pypa/setuptools/pull/2822/files#diff-e3d3d454fa3a072c9f46f8affa27513892fbc2d245e87f57a5927a7be851de05L2134-R2083

Expected behavior

When there are multiple versions of a dist present under a path entry, pkg_resources should prefer the newer one, and a call to WorkingSet.require should activate it successfully.

How to Reproduce

def test_version_priority(tmpdir):
    """
    The best version of a package should be usable.
    """
    egg_info = tmpdir / 'proj-24.egg-info'
    egg_info.mkdir()
    pkg_info = egg_info / 'PKG-INFO'
    pkg_info.write_text(
        f'Metadata-Version: 1.0\nName: proj\nVersion: 24\n',
        encoding='utf-8')

    egg_info = tmpdir / 'proj-42.egg-info'
    egg_info.mkdir()
    pkg_info = egg_info / 'PKG-INFO'
    pkg_info.write_text(
        f'Metadata-Version: 1.0\nName: proj\nVersion: 42\n',
        encoding='utf-8')

    ws = pkg_resources.WorkingSet([tmpdir])
    ws.require('proj==42')

Output

Where the reproducer passed before #2822, current HEAD raises:

pkg_resources.VersionConflict: (proj 24 (/tmp/pytest-of-cottsay/pytest-60/popen-gw0/test_version_priority0), Requirement.parse('proj==42'))

Start by running the provided test_version_priority reproducer, then inspect WorkingSet.require, WorkingSet._build_master, and WorkingSet._build_from_requirements, along with the order change referenced in pull request #2822. Done means the newer distribution is selected when multiple versions share a path entry and the requirement for version 42 succeeds without VersionConflict.