Support for "soft" or suggested constraints

Feature proposed: constraints

It would be great is Poetry supports the fantastic feature that pip has: constraints.

Constraints are extremely useful for more complex applications that have many extras - thus optional dependencies and transitive dependencies as well. It is a great tool to provide reproducible installs of Python applications, without imposing strict pinning of dependencies and allowing the users of applications to manually upgrade and downgrade dependencies of the main "application" installed, even if they are relased after the main application has been released.

Short summary of how constraints work

When installing an application in pip user can specify --constraint flag with specification of constraints to use (in the same form as requirements - local file, http URL etc.). The constraints specified this way should be "pinned" versions (i.e ==VERSION only") and they change package resolution in the way that the only the version specified for the package is considered during dependency resolution.

Constraints are not "requirements" - if the user does not install specific requirement (for example because it is part of an optional extra), the package will not be installed even if it's specific version is specified in the constraint file. Also constraints are exclusively used to perform resolution when the "installation" process resolve packages, and they are immediately forgotten once this particular "install" command completes. This allows to manually upgrade any of the packages that were "pinned" by constraints as long as it is within "requirements" specified by other packages.

Why it is useful

It is useful to get reproducible installs of applications (not libraries) without limiting security upgrades (and non-security upgrades as well).

It allows for fully reproducible, yet secure "from the scratch" Python application installs (web apps, CLI apps - generally apps that are supposed to provide user-facing features rather than libraries for other apps) without pinning specific version of dependencies in "hard" way. Fully reproducible install means that no matter if you install it today or few years from now, the application should install correctly - no matter if direect or transitive dependencies released new versions.

Typically applications might pin their dependencies to specific version and this is how you typically approach "applications" (as opposed to libraries that typically have "open" dependencies) . If you want to have truly "reproducible" install, you need to pin all your dependencies this way (including transitive ones), because otherwise transitive dependencies might break your "from the scratch" install - impacting the "first contact" with your application.

However there is a drawback of that - because when you pin dependencies, user cannot - independently - upgrade any of the dependencies that are pinned - and if those dependencies release even a small security fix, the main applicaiton must be upgraded to take into account. This is a limitation of pinning. It means that user who wants to upgrade security fix must wait for main application to release new version. In the world where supply chain attacks are a thing, and where security becomes more and more important, giving the user option to upgrade independently dependencies after the fact of installing an application is crucial.

Constraints nicely allow to make "reproducible installs" while keeping the possibility of "security updates" for any dependencies.

Another consequences of using constraints is that it also allows the user of application to perform non-security updates for the dependencies, which is important in cases like Airflow, where Airlfow is not only application to run, but also is a platform which provides library for Python developers (in Airflow DAG Authors develop workflows as Python code and they often want to be able to upgrade libraries installed by Airflow).

Lack of support for constraints is the reason why Airlfow discourages usage of poetry (even though we woudl love to be able to get their users to use poetry).

Example

Apache Airlfow is heavily depending on constraints - they developed a mechanism to automatically upgrade their constraint files based on result of automated tests and the only "recommended" way of installing airflow is via pip with constraints. Lack of constraint support is the only reason why poetry is discouraged: https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html

The constraints of airflow are maintained automatically, tagged together with each version of Airflow - main version here: https://github.com/apache/airflow/tree/constraints-main.

Yes, Airlfow is a bit special case with almost 700 dependencies and ~100 extras which is a bit extreme, but many otther applications could benefit from that approach. Constraint mechanism in Airlfow is used for more than 3 years and it helped Airflow maintainers in numerous cases where 3rd-party dependencies would have broken Airlfow "clean install" for already relased historical packages, while allowing their users to upgrade many dependencies as needed. More elaboration of why it is needed and what problems it solved for Airflow are explained in this talk from PyWaw # 98 "Managing Python dependencies at scale" https://www.youtube.com/watch?v=mlOkkTuucSk

Alternatives

As @dimbleby mentioned in https://github.com/python-poetry/poetry/issues/7047#issuecomment-1317833270 the https://github.com/python-poetry/poetry/issues/3225 closing reason was that it is the same as lock file.

While lock files are "almost" constraints, there is one difference, Lock files are development feature for someone who develops the application and once the package is uploaded to PyPI, the lock file remains only in the source code of the application. On the other hand, constraints are user facing feature. Users should be able to install an application from pypi (or another compliant repository) and apply constraints (for example taken from a published .lock file or file following requirements.txt format) as a single installation command similar to what Airlfow installation does with pip:

pip install "apache-airflow[celery]==2.4.3" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.4.3/constraints-3.7.txt"

The above is just an example, there might be other conventions used by other applications.

The convention should allow for different set constraints per Python MAJOR.MINOR verison (necessarily for big applications, set of dependencies slightly differ between Python 3.7, 3.8, etc. It would also be great to add other options - such as architecture (ARM/x86) but this is not as important as Python versions. This could be done per convention of the location of the file, or in case of poetry.lock it might be defined using poetry.lock features.

It should be possible to use lock files (and ideally also format compatible with requirements.txt) as "constraints" while installing a package by the user from PyPI, without having sources, pyproject.toml nor without having to copy poetry.lock manually to the local folder. In this sense, poetry.lock is not far from constraints, what is lacking is support for single-line installation, where constraints are specified as a remote URL to pull automatically and use durig installation by the end user from PyPI or another registry. In fact using and publishing poetry.lock as the "constraint" file to be used could be one of the main use-cases for poetry-managed applications.

Ideal properties of the constraints feature

• should be possible to run poetry install NNN --constraint http://..... to apply constraints remotely
• it should support both poetry.lock and traditional "requirements.txt" format to ease interoperabiliyt with pip
• ideally there shoudl be a tool that could allow conversion between the lock and requirement.txt

If there is a consensus among poetry maintainers that this is a worthy feature, I am happy to help in both design and implementation of this. I have vast experience in managing dependencies in Airlfow with using constraints (I am the original author of the approach Airlfow uses and I evolved and maintained it over last 3 years or so).

• I have searched the issues of this repo and believe that this is not a duplicate.

Actually there was a duplicate https://github.com/python-poetry/poetry/issues/3225 but it has been closed, but in the dicusson in https://github.com/python-poetry/poetry/issues/7047 I decided to open it againt.

• I have searched the FAQ and general documentation and believe that my question is not already covered.

Start with the existing poetry install workflow and its handling of poetry.lock; compare the requested remote constraint input with the requirements.txt format and pip's constraint behavior described here. Done means a documented installation path can apply remote lock or requirements-style constraints without the application source files, while preserving optional-dependency behavior.