AcademySoftwareFoundation / AcademySoftwareFoundation/rez
Schema - VersionRange does not support ! (not) operator
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 374
- Avg merge
- 9d 12h
- Merged PRs (30d)
- 5
Description
## Description
```rez-pip``` seems to be unable to install packages that contain the not (!) operator in their version range.
This issue has been confirmed with the popular package ```requests``` that defines its [requirements](https://github.com/kennethreitz/requests/blob/0b6c110a5cd65685c42702b27e2872ad9c234ab8/setup.py#L44) as follows:
```
requires = [
'chardet>=3.0.2,<3.1.0',
'idna>=2.5,<2.9',
'urllib3>=1.21.1,<1.26,!=1.25.0,!=1.25.1',
'certifi>=2017.4.17'
]
```
Although [rez already supports the not operator](https://github.com/nerdvegas/rez/wiki/Basic-Concepts#the-conflict-operator) during package requirements definition as specified in the wiki the syntax is different as such the pip requirement is not correctly translated to version parse-able by rez. Notably the the not operator paired with an equal sign (!=) might be the issue.
>The Conflict Operator
>The '!' operator is called the conflict operator, and is used to define an incompatibility between >packages, or to specify that you do not want a package version present. For example, consider the >command:
>]$ rez-env maya_utils '!maya-2015.6'
>This specifies that you require any version of maya_utils, but that any version of maya within 2015.6 >(and this includes 2015.6.1 and so on) is not acceptable.
### Stacktrace
#### Using pep-517
```
08:13:26 ERROR PackageMetadataError: Validation of key 'requires' failed: Or(, [And(, Use())]) did not validate [u'chardet-<3.1.0,>=3.0.2', u'idna-<2.9,>=2.5', u'urllib3-!=1.25.0,!=1.25.1,<1.26,>=1.21.1', u'certifi->=2017.4.17']
Or(And(, Use())) did not validate u'urllib3-!=1.25.0,!=1.25.1,<1.26,>=1.21.1'
PackageRequest(u'urllib3-!=1.25.0,!=1.25.1,<1.26,>=1.21.1') raised VersionError(u"Syntax error in version range '!=1.25.0,!=1.25.1,<1.26,>=1.21.1': Syntax error in version range '!=1.25.0,!=1.25.1,<1.26,>=1.21.1'",)
```
#### Using old setup.py method (ie not related to rez-pip new method)
```
Collecting requests
Downloading https://files.pythonhosted.org/packages/01/62/ddcf76d1d19885e8579acb1b1df26a852b03472c0e46d2b959a714c90608/requests-2.22.0.tar.gz (113kB)
|████████████████████████████████| 122kB 2.6MB/s
Collecting chardet<3.1.0,>=3.0.2 (from requests)
Downloading https://files.pythonhosted.org/packages/fc/bb/a5768c230f9ddb03acc9ef3f0d4a3cf93462473795d18e9535498c8f929d/chardet-3.0.4.tar.gz (1.9MB)
|████████████████████████████████| 1.9MB 27.5MB/s
Collecting idna<2.9,>=2.5 (from requests)
Downloading https://files.pythonhosted.org/packages/ad/13/eb56951b6f7950cadb579ca166e448ba77f9d24efc03edd7e55fa57d04b7/idna-2.8.tar.gz (174kB)
|████████████████████████████████| 184kB 20.1MB/s
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests)
Downloading https://files.pythonhosted.org/packages/4c/13/2386233f7ee40aa8444b47f7463338f3cbdf00c316627558784e3f542f07/urllib3-1.25.3.tar.gz (262kB)
|████████████████████████████████| 266kB 42.9MB/s
Collecting certifi>=2017.4.17 (from requests)
Downloading https://files.pythonhosted.org/packages/c5/67/5d0548226bcc34468e23a0333978f0e23d28d0b3f0c71a151aef9c3f7680/certifi-2019.6.16.tar.gz (156kB)
|████████████████████████████████| 163kB 23.2MB/s
Installing collected packages: chardet, idna, urllib3, certifi, requests
Running setup.py install for chardet ... done
Running setup.py install for idna ... done
Running setup.py install for urllib3 ... done
Running setup.py install for certifi ... done
Running setup.py install for requests ... done
Successfully installed certifi chardet idna requests urllib3
08:47:13 DEBUG Found /tmp/pip-diqKXK-rez/rez_staging/python/certifi-2019.6.16-py2.7.egg-info
08:47:13 DEBUG Found /tmp/pip-diqKXK-rez/rez_staging/python/chardet-3.0.4-py2.7.egg-info
08:47:13 DEBUG Found /tmp/pip-diqKXK-rez/rez_staging/python/idna-2.8-py2.7.egg-info
08:47:13 DEBUG Found /tmp/pip-diqKXK-rez/rez_staging/python/requests-2.22.0-py2.7.egg-info
08:47:13 WARNING Not recognised as a requirement: u''
08:47:13 WARNING Unexpected line: quitting requirement scan: u'[security]'
08:47:13 DEBUG Found /tmp/pip-diqKXK-rez/rez_staging/python/urllib3-1.25.3-py2.7.egg-info
08:47:13 WARNING Not recognised as a requirement: u''
08:47:13 WARNING Unexpected line: quitting requirement scan: u'[brotli]'
08:47:13 ERROR PackageMetadataError: Validation of key 'requires' failed: Or(, [And(, Use())]) did not validate [u'chardet-<3.1.0>=3.0.2', u'idna-<2.9>=2.5', u'urllib3-!=1.25.0!=1.25.1<1.26>=1.21.1', u'certifi->=2017.4.17']
Or(And(, Use())) did not validate u'chardet-<3.1.0>=3.0.2'
PackageRequest(u'chardet-<3.1.0>=3.0.2') raised VersionError(u"Syntax error in version range '<3.1.0>=3.0.2': Syntax error in version range '<3.1.0>=3.0.2'",) different issue??
```
#### Modified range string to only include not operator causes the issue
```
('range_str: ', '2')
('range_str: ', u'<3.1.0,>=3.0.2')
('range_str: ', u'<2.9,>=2.5')
('range_str: ', u'!=1.25.0,!=1.25.1,<1.26,>=1.21.1')
#### modified
('range_str: ', u'!=1.25.0')
('key: ', 'requires')
('attr: ', [u'chardet-<3.1.0,>=3.0.2', u'idna-<2.9,>=2.5', u'urllib3-!=1.25.0,!=1.25.1,<1.26,>=1.21.1', u'certifi->=2017.4.17'])
('schema_: ', Schema(Or(, [And(, Use())])))
Traceback (most recent call last):
File "/home/iakritas/.rez/bin/rez/rez-pip", line 4, in
run('pip')
File "/home/iakritas/.rez/lib/python2.7/site-packages/rez-2.36.0-py2.7.egg/rez/cli/_main.py", line 148, in run
returncode = run_cmd()
File "/home/iakritas/.rez/lib/python2.7/site-packages/rez-2.36.0-py2.7.egg/rez/cli/_main.py", line 140, in run_cmd
return opts.func(opts, opts.parser, extra_arg_groups)
File "/home/iakritas/.rez/lib/python2.7/site-packages/rez-2.36.0-py2.7.egg/rez/cli/pip.py", line 47, in command
release=opts.release)
File "/home/iakritas/.rez/lib/python2.7/site-packages/rez-2.36.0-py2.7.egg/rez/pip.py", line 506, in pip_install_package
pkg.commands = '\n'.join(commands)
File "/home/iakritas/.pyenv/versions/2.7.16/lib/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/home/iakritas/.rez/lib/python2.7/site-packages/rez-2.36.0-py2.7.egg/rez/package_maker__.py", line 182, in make_package
package = maker.get_package()
File "/home/iakritas/.rez/lib/python2.7/site-packages/rez-2.36.0-py2.7.egg/rez/package_maker__.py", line 125, in get_package
package.validate_data()
File "/home/iakritas/.rez/lib/python2.7/site-packages/rez-2.36.0-py2.7.egg/rez/utils/resources.py", line 266, in validate_data
self.resource.validate_data()
File "/home/iakritas/.rez/lib/python2.7/site-packages/rez-2.36.0-py2.7.egg/rez/utils/data_utils.py", line 534, in func
self.validated_data()
File "/home/iakritas/.rez/lib/python2.7/site-packages/rez-2.36.0-py2.7.egg/rez/utils/data_utils.py", line 543, in func
d[key] = getattr(self, key)
File "/home/iakritas/.rez/lib/python2.7/site-packages/rez-2.36.0-py2.7.egg/rez/utils/data_utils.py", line 183, in __get__
result = self.func(instance)
File "/home/iakritas/.rez/lib/python2.7/site-packages/rez-2.36.0-py2.7.egg/rez/utils/data_utils.py", line 585, in getter
return self._validate_key_impl(key, attr, key_schema)
File "/home/iakritas/.rez/lib/python2.7/site-packages/rez-2.36.0-py2.7.egg/rez/utils/data_utils.py", line 562, in func
return schema_.validate(attr)
File "/home/iakritas/.rez/lib/python2.7/site-packages/rez-2.36.0-py2.7.egg/rez/vendor/schema/schema.py", line 235, in validate
raise SchemaError([None] + x.autos, [e] + x.errors)
rez.vendor.schema.schema.SchemaError: Or(, [And(, Use())]) did not validate [u'chardet-<3.1.0,>=3.0.2', u'idna-<2.9,>=2.5', u'urllib3-!=1.25.0,!=1.25.1,<1.26,>=1.21.1', u'certifi->=2017.4.17']
Or(And(, Use())) did not validate u'urllib3-!=1.25.0,!=1.25.1,<1.26,>=1.21.1'
PackageRequest(u'urllib3-!=1.25.0,!=1.25.1,<1.26,>=1.21.1') raised VersionError(u"Syntax error in version range '!=1.25.0': Syntax error in version range '!=1.25.0'",)
```
#### Removing the not parts from the range correctly installs the package
```
'range_str: ', '')
('range_str: ', '2.7')
('range_str: ', '==linux')
('range_str: ', '==x86_64')
('range_str: ', '==Arch-rolling')
('range_str: ', 'linux')
('range_str: ', 'x86_64')
('range_str: ', 'Arch-rolling')
('range_str: ', '2.7')
('range_str: ', 'linux')
('range_str: ', 'x86_64')
('range_str: ', 'Arch-rolling')
('range_str: ', 'linux')
('range_str: ', 'x86_64')
11:39:47 INFO Using pip-19.1.1 (/home/iakritas/packages/pip/19.1.1/package.py[0])
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting requests
Using cached https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests)
Using cached https://files.pythonhosted.org/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
Using cached https://files.pythonhosted.org/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests)
Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Installing collected packages: urllib3, certifi, chardet, idna, requests
Successfully installed certifi-2019.6.16 chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.3
11:39:49 DEBUG Found /tmp/pip-2JhFhg-rez/rez_staging/python/certifi-2019.6.16.dist-info
11:39:49 DEBUG Found /tmp/pip-2JhFhg-rez/rez_staging/python/chardet-3.0.4.dist-info
11:39:49 DEBUG Removed 1.0 due to Requires-Python
11:39:49 DEBUG Removed 1.1 due to Requires-Python
11:39:49 DEBUG Found /tmp/pip-2JhFhg-rez/rez_staging/python/idna-2.8.dist-info
11:39:49 DEBUG Removed 1.0 due to Requires-Python
11:39:49 DEBUG Removed 1.1 due to Requires-Python
11:39:49 DEBUG Removed 1.2 due to Description-Content-Type
11:39:49 DEBUG Removed 2.0 due to Description-Content-Type
11:39:49 DEBUG Found /tmp/pip-2JhFhg-rez/rez_staging/python/requests-2.22.0.dist-info
11:39:49 DEBUG Removed 1.0 due to Requires-Python
11:39:49 DEBUG Removed 1.1 due to Requires-Python
11:39:49 DEBUG Removed 1.2 due to Provides-Extra
11:39:49 DEBUG Found /tmp/pip-2JhFhg-rez/rez_staging/python/urllib3-1.25.3.dist-info
('range_str: ', '2')
('range_str: ', '2')
('range_str: ', '2')
('range_str: ', u'<3.1.0,>=3.0.2')
('range_str: ', u'<2.9,>=2.5')
('range_str: ', u'!=1.25.0,!=1.25.1,<1.26,>=1.21.1')
#### modified
('range_str: ', u'<1.26,>=1.21.1')
('range_str: ', u'>=2017.4.17')
('range_str: ', '2')
('range_str: ', '2')
('range_str: ', '2')
('range_str: ', '2')
('range_str: ', '2')
('range_str: ', '2')
('range_str: ', '2')
5 packages were installed:
idna-2.8: /home/iakritas/packages/idna/2.8/package.py (python-2)
requests-2.22.0: /home/iakritas/packages/requests/2.22.0/package.py (python-2)
urllib3-1.25.3: /home/iakritas/packages/urllib3/1.25.3/package.py (python-2)
certifi-2019.6.16: /home/iakritas/packages/certifi/2019.6.16/package.py (python-2)
chardet-3.0.4: /home/iakritas/packages/chardet/3.0.4/package.py (python-2)
```
Contributor guide
Research direction
Start with rez/cli/pip.py and the PackageRequest entry point in rez/utils/formatting, using the requests requirement containing != as the reproduction. Run rez-pip with that requirement and verify that the not operator is accepted without the reported VersionError or package metadata validation failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100