Requirements for Python dependency management

Hi @web-platform-tests/wpt-core-team!

Recently I have taken a closer look at how we track and install Python dependencies here in WPT. I wrote up two fairly detailed issues about what we're currently doing in https://github.com/web-platform-tests/wpt/issues/28801 and https://github.com/web-platform-tests/wpt/issues/28809.

I suspect that there's room to improve matters here, but want to gather requirements before I suggest something. But first, some problem I think are worth trying to solve:

• Even though we vendor a lot of things, it's not enough to use ./wpt run (virtualenv is the additional dependency)
• There's overlap between vendored and installed dependencies (e.g. aioquic, html5lib, pytest) and it's non-obvious if different versions are a problem or just fine
• We don't pin all transient dependencies, which can (rarely) cause sudden breakage like in https://github.com/web-platform-tests/wpt/pull/28570
• Updating tools/third_party/ is tricky, involving git subtree, and can't be merged like regular wpt PRs

Here are the requirements I would consider for any change here, roughly in order of importance:

• Downstream users (Chromium, Gecko, WebKit) must be able to manage dependencies separately, avoiding any vendored or automatically downloaded dependencies in this repo. It should be easy to know which versions are used in wpt, but possible to diverge from that if necessary.
• Users of ./wpt run should need at most one tool in addition to Python (currently Virtualenv) but zero is even better.
• The distinction between direct and indirect dependencies should be clear, to avoid orphaned (unused) dependencies.
• All dependencies, direct and indirect, should be pinned (think package-lock.json or Pipfile.lock)
• Dependency updates should still be infrequent and deliberate, since they cause downstream work.
• Installing dependencies and checking if they need to be updated should be fast.

I'm interested to hear if others have had trouble with dependencies, and if there are additional constraints/requirements here.

Start by reading the linked issues 28801 and 28809, then inspect the current dependency handling around ./wpt run and tools/third_party/. The issue is gathering requirements rather than naming an implementation target; done would require an agreed dependency-management design that satisfies the listed downstream, pinning, update, and installation constraints.