pathological version solving? (django-distill + boto3)

• Poetry version: 1.7.1
• Python version: 3.11
• OS version and name: macOS 14.1.1
• pyproject.toml:

[tool.poetry]
name = "poetrytemp"

[tool.poetry.dependencies]
python = "^3.11"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

I can reproduce this in an empty poetry project.

1. attempt to poetry add -vv 'django-distill[amazon]'

output is like:

Resolving dependencies...
   1: fact: poetrytemp is 0.1.0
   1: derived: poetrytemp
   1: fact: poetrytemp depends on django-distill (^3.1.3)
   1: selecting poetrytemp (0.1.0)
   1: derived: django-distill[amazon] (>=3.1.3,<4.0.0)
   1: fact: django-distill (3.1.3) depends on django-distill (3.1.3)
   1: fact: django-distill (3.1.3) depends on django (*)
   1: fact: django-distill (3.1.3) depends on requests (*)
   1: fact: django-distill (3.1.3) depends on boto3 (*)
   1: selecting django-distill[amazon] (3.1.3)
   1: derived: boto3
   1: derived: requests
   1: derived: django
   1: derived: django-distill (==3.1.3)
   1: fact: django-distill (3.1.3) depends on django (*)
   1: fact: django-distill (3.1.3) depends on requests (*)
   1: selecting django-distill (3.1.3)
   1: fact: django (5.0) depends on asgiref (>=3.7.0)
   1: fact: django (5.0) depends on sqlparse (>=0.3.1)
   1: fact: django (5.0) depends on tzdata (*)
   1: selecting django (5.0)
   1: derived: tzdata
   1: derived: sqlparse (>=0.3.1)
   1: derived: asgiref (>=3.7.0)
   1: fact: requests (2.31.0) depends on charset-normalizer (>=2,<4)
   1: fact: requests (2.31.0) depends on idna (>=2.5,<4)
   1: fact: requests (2.31.0) depends on urllib3 (>=1.21.1,<3)
   1: fact: requests (2.31.0) depends on certifi (>=2017.4.17)
   1: selecting requests (2.31.0)
   1: derived: certifi (>=2017.4.17)
   1: derived: urllib3 (>=1.21.1,<3)
   1: derived: idna (>=2.5,<4)
   1: derived: charset-normalizer (>=2,<4)
   1: selecting urllib3 (2.1.0)
   1: selecting certifi (2023.11.17)
   1: selecting charset-normalizer (3.3.2)
   1: selecting idna (3.6)
   1: selecting sqlparse (0.4.4)
   1: selecting asgiref (3.7.2)
   1: fact: boto3 (1.34.7) depends on botocore (>=1.34.7,<1.35.0)
   1: fact: boto3 (1.34.7) depends on jmespath (>=0.7.1,<2.0.0)
   1: fact: boto3 (1.34.7) depends on s3transfer (>=0.10.0,<0.11.0)
   1: selecting boto3 (1.34.7)
   1: derived: s3transfer (>=0.10.0,<0.11.0)
   1: derived: jmespath (>=0.7.1,<2.0.0)
   1: derived: botocore (>=1.34.7,<1.35.0)
   1: fact: s3transfer (0.10.0) depends on botocore (>=1.33.2,<2.0a.0)
   1: selecting s3transfer (0.10.0)
   1: fact: botocore (1.34.7) depends on jmespath (>=0.7.1,<2.0.0)
   1: fact: botocore (1.34.7) depends on python-dateutil (>=2.1,<3.0.0)
   1: fact: botocore (1.34.7) depends on urllib3 (>=1.25.4,<2.1)
   1: derived: not botocore (==1.34.7)
   1: fact: no versions of botocore match >1.34.7,<1.35.0
   1: conflict: no versions of botocore match >1.34.7,<1.35.0
   1: derived: not botocore (>1.34.7,<1.35.0)
   1: conflict: botocore (1.34.7) depends on urllib3 (>=1.25.4,<2.1)
   1: ! botocore (==1.34.7) is partially satisfied by not botocore (>1.34.7,<1.35.0)
   1: ! which is caused by "no versions of botocore match >1.34.7,<1.35.0"
   1: ! thus: botocore (>=1.34.7,<1.35.0) requires urllib3 (>=1.25.4,<2.1)
   1: fact: botocore (>=1.34.7,<1.35.0) requires urllib3 (>=1.25.4,<2.1)
   1: derived: not botocore (>=1.34.7,<1.35.0)
   1: derived: not boto3 (==1.34.7)
   2: selecting certifi (2023.11.17)
   2: selecting charset-normalizer (3.3.2)
   2: selecting idna (3.6)
   2: selecting sqlparse (0.4.4)
   2: selecting asgiref (3.7.2)
   2: fact: boto3 (1.34.6) depends on botocore (>=1.34.6,<1.35.0)
   2: fact: boto3 (1.34.6) depends on jmespath (>=0.7.1,<2.0.0)
   2: fact: boto3 (1.34.6) depends on s3transfer (>=0.10.0,<0.11.0)
   2: selecting boto3 (1.34.6)
   2: derived: s3transfer (>=0.10.0,<0.11.0)
   2: derived: jmespath (>=0.7.1,<2.0.0)
   2: derived: botocore (>=1.34.6,<1.35.0)
   2: fact: s3transfer (0.10.0) depends on botocore (>=1.33.2,<2.0a.0)
   2: selecting s3transfer (0.10.0)
   2: fact: botocore (1.34.6) depends on jmespath (>=0.7.1,<2.0.0)
   2: fact: botocore (1.34.6) depends on python-dateutil (>=2.1,<3.0.0)
   2: fact: botocore (1.34.6) depends on urllib3 (>=1.25.4,<2.1)
   2: derived: not botocore (==1.34.6)
   2: fact: no versions of botocore match >1.34.6,<1.34.7
   2: conflict: no versions of botocore match >1.34.6,<1.34.7
   2: derived: not botocore (>1.34.6,<1.34.7)
   2: conflict: botocore (1.34.6) depends on urllib3 (>=1.25.4,<2.1)
   2: ! botocore (==1.34.6) is partially satisfied by not botocore (>1.34.6,<1.34.7)
   2: ! which is caused by "no versions of botocore match >1.34.6,<1.34.7"
   2: ! thus: botocore (>=1.34.6,<1.34.7) requires urllib3 (>=1.25.4,<2.1)

this continues forever in a fruitless search, downloading the wheel for every minor version of botocore

the problem seems to be something to do with urllib3

   1: fact: requests (2.31.0) depends on urllib3 (>=1.21.1,<3)

   1: derived: urllib3 (>=1.21.1,<3)

   1: selecting urllib3 (2.1.0)

   1: fact: botocore (1.34.7) depends on urllib3 (>=1.25.4,<2.1)

   1: conflict: botocore (1.34.7) depends on urllib3 (>=1.25.4,<2.1)
   1: ! botocore (==1.34.7) is partially satisfied by not botocore (>1.34.7,<1.35.0)
   1: ! which is caused by "no versions of botocore match >1.34.7,<1.35.0"
   1: ! thus: botocore (>=1.34.7,<1.35.0) requires urllib3 (>=1.25.4,<2.1)
   1: fact: botocore (>=1.34.7,<1.35.0) requires urllib3 (>=1.25.4,<2.1)

it's not clear from the output why https://pypi.org/project/urllib3/1.26.18/ couldn't be used to satisfy >=1.25.4,<2.1

Ok but then the odd thing is if I do this:

1. poetry add boto3
2. poetry add 'django-distill[amazon]'

then it works fine!

If we look at the setup.py for django-distill: https://github.com/meeb/django-distill/blob/v3.1.3/setup.py#L30
it specifies:

    extras_require = {
        'amazon': ['boto3'],

So no version constraint on boto3.

It doesn't make much sense that dependency resolution fails one way but not the other?

Reproduce the issue from the provided pyproject.toml with poetry add -vv 'django-distill[amazon]', then compare it with adding boto3 first. Inspect the dependency-resolution path and the django-distill setup.py extras_require entry; done means the resolver handles both install orders consistently without the fruitless botocore search.