1.18.50+ breaks minor version pinning with boto3 & botocore
- Dominant language
- Python
- Stars
- 17.3k
- Forks
- 4.6k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 13
Description
Confirm by changing [ ] to [x] below to ensure that it's a bug:
- [x] I've gone though the [User Guide](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) and the [API reference](https://docs.aws.amazon.com/cli/latest/reference/)
- [x] I've searched for [previous similar issues](https://github.com/aws/aws-cli/issues) and didn't find any solution
**Describe the bug**
Unable to satisfy dependencies when pinning awscli, boto3 & botocore minor versions.
EA, the below `setup.cfg` snippet causes dependents to fail pkg_resource dependency check:
```
install_requires =
awscli >= 1.18.34, <1.19
boto3 >= 1.12.34, <1.13
botocore >= 1.15.34, <1.16
```
**SDK version number**
1.18.50+
**Platform/OS/Hardware/Device**
Linux/macos/windows
**To Reproduce (observed behavior)**
The dependency error can be reproduced by running the below install command. `aws` cli still works. But any python package with the above requirements will have its cli scripts fail as later.
```
$ pip install 'awscli<1.19' 'boto3<1.13' 'botocore<1.16'
...
ERROR: awscli 1.18.51 has requirement botocore==1.16.1, but you'll have botocore 1.15.49 which is incompatible.
...
```
**Expected behavior**
Installation success, dependant cli unaffected.
**Logs/output**
```
$ pip install -e .
Obtaining file:///Users/gabdav01/work/paas/splatt
Collecting appdirs<1.5,>=1.4.3
Using cached appdirs-1.4.3-py2.py3-none-any.whl (12 kB)
Collecting awscli<1.19,>=1.18.34
Using cached awscli-1.18.51-py2.py3-none-any.whl (3.0 MB)
Collecting boto3<1.13,>=1.12.34
Using cached boto3-1.12.49-py2.py3-none-any.whl (128 kB)
Collecting botocore<1.16,>=1.15.34
Using cached botocore-1.15.49-py2.py3-none-any.whl (6.2 MB)
Collecting colorama<0.4.4
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Processing /Users/gabdav01/Library/Caches/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-macosx_10_14_x86_64.whl
Processing /Users/gabdav01/Library/Caches/pip/wheels/e4/27/61/1cdc0b98cfda1ba5fc6d97cb7519f083427323ec7d191bfc0e/dpath-1.5.0-cp37-none-any.whl
Collecting click==7.1.1
Using cached click-7.1.1-py2.py3-none-any.whl (82 kB)
Collecting click-log==0.3.2
Using cached click_log-0.3.2-py2.py3-none-any.whl (4.6 kB)
Collecting cryptography<2.10,>=2.9
Using cached cryptography-2.9.2-cp35-abi3-macosx_10_9_x86_64.whl (1.8 MB)
Collecting dnspython==1.16.0
Using cached dnspython-1.16.0-py2.py3-none-any.whl (188 kB)
Processing /Users/gabdav01/Library/Caches/pip/wheels/ea/32/d0/77864057ba71ebce6ab3a8eda50c11971b005c12fa059eebde/f5_sdk-3.0.21-cp37-none-any.whl
Collecting Jinja2<2.12,>=2.11.2
Using cached Jinja2-2.11.2-py2.py3-none-any.whl (125 kB)
Collecting kubernetes[adal]==11.0.0
Using cached kubernetes-11.0.0-py3-none-any.whl (1.5 MB)
Collecting paramiko<2.8,>=2.7.1
Using cached paramiko-2.7.1-py2.py3-none-any.whl (206 kB)
Collecting passlib==1.7.2
Using cached passlib-1.7.2-py2.py3-none-any.whl (507 kB)
Collecting pyjks<20,>=19.0.0
Using cached pyjks-19.0.0-py2.py3-none-any.whl (45 kB)
Collecting pyjwt<1.8,>=1.7.1
Using cached PyJWT-1.7.1-py2.py3-none-any.whl (18 kB)
Collecting pyopenssl<20.0.0,>=19.0.0
Using cached pyOpenSSL-19.1.0-py2.py3-none-any.whl (53 kB)
Collecting semantic_version<2.9,>=2.8.2
Downloading semantic_version-2.8.5-py2.py3-none-any.whl (15 kB)
Collecting tenacity<6.2,>=6.1
Using cached tenacity-6.1.0-py2.py3-none-any.whl (24 kB)
Collecting requests<2.24,>=2.23
Using cached requests-2.23.0-py2.py3-none-any.whl (58 kB)
Collecting urllib3<1.26,>=1.25.6
Using cached urllib3-1.25.9-py2.py3-none-any.whl (126 kB)
Collecting idna<3,>=2.9
Using cached idna-2.9-py2.py3-none-any.whl (58 kB)
Collecting rsa<=3.5.0,>=3.1.2
Using cached rsa-3.4.2-py2.py3-none-any.whl (46 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.9.5-py2.py3-none-any.whl (24 kB)
Collecting python-dateutil<3.0.0,>=2.1
Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting cffi!=1.11.3,>=1.8
Using cached cffi-1.14.0-cp37-cp37m-macosx_10_9_x86_64.whl (174 kB)
Collecting six>=1.4.1
Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Processing /Users/gabdav01/Library/Caches/pip/wheels/12/43/4c/9c3051759105429cd633ed53ece367db44b082fce01c28db46/f5_icontrol_rest-1.3.13-cp37-none-any.whl
Collecting MarkupSafe>=0.23
Using cached MarkupSafe-1.1.1-cp37-cp37m-macosx_10_6_intel.whl (18 kB)
Collecting google-auth>=1.0.1
Using cached google_auth-1.14.1-py2.py3-none-any.whl (89 kB)
Collecting requests-oauthlib
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Requirement already satisfied: setuptools>=21.0.0 in /Users/gabdav01/.virtualenvs/test-splatt-install/lib/python3.7/site-packages (from kubernetes[adal]==11.0.0->splatt==1.16.5) (46.1.3)
Collecting certifi>=14.05.14
Using cached certifi-2020.4.5.1-py2.py3-none-any.whl (157 kB)
Collecting adal>=1.0.2; extra == "adal"
Downloading adal-1.2.3-py2.py3-none-any.whl (53 kB)
|████████████████████████████████| 53 kB 522 kB/s
Collecting pynacl>=1.0.1
Using cached PyNaCl-1.3.0-cp34-abi3-macosx_10_6_intel.whl (284 kB)
Collecting bcrypt>=3.1.3
Using cached bcrypt-3.1.7-cp34-abi3-macosx_10_6_intel.whl (53 kB)
Processing /Users/gabdav01/Library/Caches/pip/wheels/e9/ce/00/66bd8b019c093719e2d78ea39a6f80c8133aecc1ec3f77f41c/twofish-0.3.0-cp37-cp37m-macosx_10_14_x86_64.whl
Collecting javaobj-py3
Using cached javaobj_py3-0.4.1-py2.py3-none-any.whl (67 kB)
Collecting pyasn1>=0.3.5
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting pyasn1-modules
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting pycryptodomex
Using cached pycryptodomex-3.9.7-cp37-cp37m-macosx_10_6_intel.whl (14.1 MB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Collecting cachetools<5.0,>=2.0.0
Using cached cachetools-4.1.0-py3-none-any.whl (10 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.0-py2.py3-none-any.whl (147 kB)
ERROR: awscli 1.18.51 has requirement botocore==1.16.1, but you'll have botocore 1.15.49 which is incompatible.
Installing collected packages: appdirs, pyasn1, rsa, PyYAML, docutils, urllib3, six, python-dateutil, jmespath, botocore, colorama, s3transfer, awscli, boto3, dpath, click, click-log, pycparser, cffi, cryptography, dnspython, idna, chardet, certifi, requests, f5-icontrol-rest, f5-sdk, MarkupSafe, Jinja2, cachetools, pyasn1-modules, google-auth, oauthlib, requests-oauthlib, websocket-client, pyjwt, adal, kubernetes, pynacl, bcrypt, paramiko, passlib, twofish, javaobj-py3, pycryptodomex, pyjks, pyopenssl, semantic-version, tenacity, splatt
Running setup.py develop for splatt
Successfully installed Jinja2-2.11.2 MarkupSafe-1.1.1 PyYAML-5.3.1 adal-1.2.3 appdirs-1.4.3 awscli-1.18.51 bcrypt-3.1.7 boto3-1.12.49 botocore-1.15.49 cachetools-4.1.0 certifi-2020.4.5.1 cffi-1.14.0 chardet-3.0.4 click-7.1.1 click-log-0.3.2 colorama-0.4.3 cryptography-2.9.2 dnspython-1.16.0 docutils-0.15.2 dpath-1.5.0 f5-icontrol-rest-1.3.13 f5-sdk-3.0.21 google-auth-1.14.1 idna-2.9 javaobj-py3-0.4.1 jmespath-0.9.5 kubernetes-11.0.0 oauthlib-3.1.0 paramiko-2.7.1 passlib-1.7.2 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pycryptodomex-3.9.7 pyjks-19.0.0 pyjwt-1.7.1 pynacl-1.3.0 pyopenssl-19.1.0 python-dateutil-2.8.1 requests-2.23.0 requests-oauthlib-1.3.0 rsa-3.4.2 s3transfer-0.3.3 semantic-version-2.8.5 six-1.14.0 splatt tenacity-6.1.0 twofish-0.3.0 urllib3-1.25.9 websocket-client-0.57.0
(test-splatt-install)
gabdav01@e109198-mac:~/work/paas/splatt on aws-alloc-eip [!?$]
$ splatt --version
Traceback (most recent call last):
File "/Users/gabdav01/.virtualenvs/test-splatt-install/lib/python3.7/site-packages/pkg_resources/__init__.py", line 584, in _build_master
ws.require(__requires__)
File "/Users/gabdav01/.virtualenvs/test-splatt-install/lib/python3.7/site-packages/pkg_resources/__init__.py", line 901, in require
needed = self.resolve(parse_requirements(requirements))
File "/Users/gabdav01/.virtualenvs/test-splatt-install/lib/python3.7/site-packages/pkg_resources/__init__.py", line 792, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (botocore 1.15.49 (/Users/gabdav01/.virtualenvs/test-splatt-install/lib/python3.7/site-packages), Requirement.parse('botocore==1.16.1'), {'awscli'})
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/gabdav01/.virtualenvs/test-splatt-install/bin/splatt", line 6, in
from pkg_resources import load_entry_point
File "/Users/gabdav01/.virtualenvs/test-splatt-install/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3258, in
@_call_aside
File "/Users/gabdav01/.virtualenvs/test-splatt-install/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3242, in _call_aside
f(*args, **kwargs)
File "/Users/gabdav01/.virtualenvs/test-splatt-install/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3271, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/Users/gabdav01/.virtualenvs/test-splatt-install/lib/python3.7/site-packages/pkg_resources/__init__.py", line 586, in _build_master
return cls._build_from_requirements(__requires__)
File "/Users/gabdav01/.virtualenvs/test-splatt-install/lib/python3.7/site-packages/pkg_resources/__init__.py", line 599, in _build_from_requirements
dists = ws.resolve(reqs, Environment())
File "/Users/gabdav01/.virtualenvs/test-splatt-install/lib/python3.7/site-packages/pkg_resources/__init__.py", line 792, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (botocore 1.15.49 (/Users/gabdav01/.virtualenvs/test-splatt-install/lib/python3.7/site-packages), Requirement.parse('botocore==1.16.1'), {'awscli'})
(test-splatt-install)
```
**Additional context**
```
pip install 'awscli==1.18.49' 'boto3<1.13' 'botocore<1.16'
```
Previously, every time a minor botocore version was changed, so did awscli. Boto3 still maintains this practice. But AWS cli failed to do so for the past 2 minor botocore version changes.
Contributor guide
Assessment
This issue has not been assessed yet.