CI checks suffer from cross-PR dependency pollution when there are git dependencies

If a branch uses a git+https://... dependency in order to integration-test some pending changes to a dependency, that version of the dependency may end up used in the CI checks for other branches, which could produce incorrect test successes or failures. We first observed this two weeks ago, but it is likely that it has been happening for longer.

These appear to be the necessary conditions:

• The same virtualenv is used for multiple branches. This can happen on a developer's machine, or when using the 2U-provided CI runners (which re-use virtualenvs).
• On one branch (such as master), a dependency is installed from PyPI, but on another branch the dependency is changed to install from git (example) and requirements are re-compiled (example)
• The git-dependency's version is the same as the one on PyPI (that is, its setup.py version value has not changed on the feature branch that is pointed to by the requirements file)
• CI checks are run for the git-using branch and then for the PyPI-using branch

This is in spite of the CI checks using make test-requirements, which calls pip-sync. Normally, this clears out all stale packages, but it fails to in this situation.

Theory

When pip-sync sees a git-dependency, it always installs it (after uninstalling any current version.) When it sees a PyPI dependency, however, it first checks to see if the version is already satisfied. So after the switch back to the base branch, pip-sync has no idea that the wrong dependency is still installed.

Unknowns

• Does this still happen if the git-dependency uses a commit hash reference rather than a branch?
• Is this a bug in pip-tools? (Should it always uninstall when there's a current git-dependency?)
• Does this still happen with newer versions of pip and pip-tools?

Reproduce

This can be reproduced by using the release-2023-02-27-10.29 tag as the base commit (standing in for master) and timmc/repro-pip-stale-git-other as the integration test branch (compare). The "other branch" uses a branch of edx-drf-extensions that has a new dependency on cryptojwt (with no version change) and code changes that depend on it.

In a completely fresh Python 3.8 virtualenv, run the following in order to install requirements for the base branch, then the other branch, then the base branch again—and check the consistency of the environment after each requirements sync. It will end up using pip 23.0.1 and pip-sync 6.12.2.

git checkout release-2023-02-27-10.29
make test-requirements
pip check
# "No broken requirements found."
python -c 'import edx_rest_framework_extensions.auth.jwt.tests.utils'
# no error


git checkout timmc/repro-pip-stale-git-other
make test-requirements
# ...
# Found existing installation: edx-drf-extensions 8.4.1
# Uninstalling edx-drf-extensions-8.4.1:
#   Successfully uninstalled edx-drf-extensions-8.4.1
# ...
# Collecting edx-drf-extensions@ git+https://github.com/openedx/edx-drf-extensions@timmc/repro-pip-stale-git
#   ...
#   Resolved https://github.com/openedx/edx-drf-extensions to commit 8719107b131cef344ae36288896b565b2a632321
# ...
# Collecting cryptojwt==1.8.3
#   Using cached cryptojwt-1.8.3-py3-none-any.whl (85 kB)
# ...
# Successfully installed cryptojwt-1.8.3 edx-drf-extensions-8.4.1 mongodbproxy-0.1.0+edx.2 olxcleaner-0.1.3
# ...
pip check
# "No broken requirements found."
python -c 'import edx_rest_framework_extensions.auth.jwt.tests.utils'
# no error


git checkout release-2023-02-27-10.29
make test-requirements
# Found existing installation: cryptojwt 1.8.3
# Uninstalling cryptojwt-1.8.3:
#  Successfully uninstalled cryptojwt-1.8.3
# ...
# [does not install/uninstall edx-drf-extensions]
pip check
# "edx-drf-extensions 8.4.1 requires cryptojwt, which is not installed."
python -c 'import edx_rest_framework_extensions.auth.jwt.tests.utils'
# Traceback (most recent call last):
#   File "<string>", line 1, in <module>
#   File "/home/timmc/edx-repos/edx-platform/venv-3.8/lib/python3.8/site-packages/edx_rest_framework_extensions/auth/jwt/tests/utils.py", line 6, in <module>
#     from cryptojwt.jws.jws import JWS
# ModuleNotFoundError: No module named 'cryptojwt'

Reproduce the reported sequence in a fresh Python 3.8 virtualenv: run make test-requirements and pip check on the base branch, integration branch, and base branch again. Inspect the make test-requirements and pip-sync behavior, including the stated pip and pip-tools versions, and compare results with newer versions. Done means switching branches no longer leaves an incorrect dependency state or produces misleading CI results.