dependabot / dependabot/dependabot-core
Support per-lockfile (pip constraints file) Python interpreter version selection
- Dominant language
- Ruby
- Stars
- 5.8k
- Forks
- 1.5k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 149
Description
Huh?
====
Python libraries/frameworks typically support a range of interpreter versions. They are tested in different environments. The Python ecosystem is rather sophisticated — the distribution packages may declare that they support one range of Python versions but not the other. Besides, some releases may ship binary distributions (wheels) targeting different platforms (OS+arch). On top of that, there's a possibility of marking some of the dependencies as conditional (usually, depending on the Python interpreter version, OS, or architecture). This results in different dependency trees produced when they are generated under different platforms/Pythons with pip-tools.
Sometimes, the generated file may be compatible with other platforms but this is not always true. I've documented one of such cases in https://github.com/dependabot/dependabot-core/issues/4606.
This is why I've got a number of pip-tools-generated files per Python version x OS x arch that are picked up depending on the current environment.
The problem is that each of those should be generated + updated in the environment that it targets. And Dependabot is currently just using only one Python version under Linux.
I suppose that adding more OSs+arch versions into the mix may be problematic but Dependabot should at least allow declaring a per-file Python version which is what this FR is about.
I've seen some attempts to infer the interpreter version from the environment markers in `constraints.txt` files but this is not enough and that matching mechanism is imperfect. In the previous issue (https://github.com/dependabot/dependabot-core/issues/4606), what's happening is that Dependabot tries updating a constraints file for Python 2.7 while using Python 3.10 and fails on the `watchdog` package which is an indirect transitive dependency that has an upper bound set to `< 3.6` in `python_requires` (https://github.com/gorakhargosh/watchdog/blob/v0.10.7/setup.py#L159) for v0.10.7 while v1.0.0 has `>= 3.6` meaning that Python 2.7-3.5 must use `watchdog < 1` and Python 3.6-3.11 must use `watchdog >= 1` — there's not a single `watchdog` release supporting the whole Python version range of `>= 2.7, <= 3.11`.
This is not a problem for downstream projects that target a single deployment environment but is a showstopper for projects that are frameworks or libraries.
Now what
========
I don't see any possibility to declare this in `dependabot.yml` but how about having a special comment in the requirements/constraints files or maybe better have that encoded in the file name. My current approach uses the latter — the base file name format I use is `---`:
```console
requirements/tox-build-dists-cp310-linux-x86_64.in
requirements/tox-build-dists-cp310-linux-x86_64.txt
requirements/tox-build-dists-cp39-linux-x86_64.in
requirements/tox-build-dists-cp39-linux-x86_64.txt
```
Another idea could be retrieving this information from comments that pip-tools outputs like:
```console
# This file is autogenerated by pip-compile with python 3.10
```
Contributor guide
Research direction
Start by reading how Dependabot handles Python requirements and constraints files, including the proposed dependabot.yml, filename conventions, and pip-tools comments described here. Review issue #4606 for the failing Python-version case; done should mean a constraints file can select its intended interpreter version during generation and updates.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100