Azure / Azure/cyclecloud-gridengine

Dependency 'Requests' Fails in Python 3.6

Open
#35 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7
Forks
18
Avg merge
5d 21h
Merged PRs (30d)
2

Description

I am attempting to use the cyclecloud-gridengine 2.0.13 release on a RHEL 7.9 scheduler VM. The yum python3 package version for this Linux distribution is 3.6.8. I am running into errors in the jetpack deployment process with Python dependencies. You can recreate the issue by running the `install.sh` line out of the `bash 'setup cyclecloud-gridengine'` block in the [scheduler.rb](https://github.com/Azure/cyclecloud-gridengine/blob/master/specs/default/chef/site-cookbooks/gridengine/recipes/scheduler.rb) recipe. Below is the error that was encountered in chef-client.log:
```
[2022-09-13T17:49:54+00:00] ERROR: bash[setup cyclecloud-gridengine] (gridengine::scheduler line 526) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20220913-26255-1e86596" ----
STDOUT: INSTALL_PYTHON3=1
INSTALL_VIRTUALENV=1
VENV=/opt/cycle/gridengine/venv
created virtual environment CPython3.6.8.final.0-64 in 346ms
creator CPython3Posix(dest=/opt/cycle/gridengine/venv, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/.local/share/virtualenv)
added seed packages: pip==21.3.1, setuptools==59.6.0, wheel==0.37.1
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Processing ./packages/certifi-2022.6.15-py3-none-any.whl
Processing ./packages/charset_normalizer-2.1.1-py3-none-any.whl
Processing ./packages/cyclecloud_api-8.0.1-py2.py3-none-any.whl
Processing ./packages/cyclecloud-gridengine-2.0.13.tar.gz
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Processing ./packages/cyclecloud-scalelib-0.1.7.tar.gz
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Processing ./packages/idna-3.3-py3-none-any.whl
Processing ./packages/immutabledict-1.0.0-py3-none-any.whl
Processing ./packages/importlib_metadata-3.6.0-py3-none-any.whl
Processing ./packages/jsonpickle-1.4.1-py2.py3-none-any.whl
Processing ./packages/requests-2.28.1-py3-none-any.whl
Processing ./packages/six-1.16.0-py2.py3-none-any.whl
Processing ./packages/typing_extensions-4.3.0-py3-none-any.whl
Processing ./packages/urllib3-1.26.12-py2.py3-none-any.whl
Processing ./packages/zipp-3.6.0-py3-none-any.whl
INFO: pip is looking at multiple versions of importlib-metadata to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of idna to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of jsonpickle to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of immutabledict to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of cyclecloud-scalelib to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of cyclecloud-gridengine to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of cyclecloud-api to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of charset-normalizer to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of certifi to determine which version is compatible with other requirements. This could take a while.
Traceback (most recent call last):
File "/opt/cycle/gridengine/venv/bin/azge", line 4, in
from gridengine.cli import main
ModuleNotFoundError: No module named 'gridengine'
STDERR: + . /etc/cluster-setup.sh
++ SGE_ROOT=/sched/sge/sge-2011-11j
++ export SGE_ROOT
+++ /sched/sge/sge-2011-11j/util/arch
++ ARCH=linux-x64
+++ /sched/sge/sge-2011-11j/util/arch -m
++ DEFAULTMANPATH=/usr/share/man/overrides:/usr/share/man:/usr/local/share/man
+++ /sched/sge/sge-2011-11j/util/arch -mt
++ MANTYPE=man
++ SGE_CELL=default
++ export SGE_CELL
++ SGE_CLUSTER_NAME=grid1
++ export SGE_CLUSTER_NAME
++ SGE_QMASTER_PORT=537
++ export SGE_QMASTER_PORT
++ SGE_EXECD_PORT=538
++ export SGE_EXECD_PORT
++ '[' '' = '' ']'
++ MANPATH=/usr/share/man/overrides:/usr/share/man:/usr/local/share/man
++ MANPATH=/sched/sge/sge-2011-11j/man:/usr/share/man/overrides:/usr/share/man:/usr/local/share/man
++ export MANPATH
++ PATH=/sched/sge/sge-2011-11j/bin/linux-x64:/opt/cycle/jetpack/system/embedded/bin:/opt/cycle/jetpack/system/bin:/opt/cycle/jetpack/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ export PATH
++ case $ARCH in
++ unset ARCH DEFAULTMANPATH MANTYPE
+ cd /opt/cycle/jetpack/system/bootstrap/
+ rm -f cyclecloud-gridengine-pkg-2.0.13.tar.gz
+ jetpack download cyclecloud-gridengine-pkg-2.0.13.tar.gz --project gridengine ./
+ tar xzf cyclecloud-gridengine-pkg-2.0.13.tar.gz
+ cd cyclecloud-gridengine/
+ INSTALLDIR=/opt/cycle/gridengine
+ mkdir -p /opt/cycle/gridengine/venv
+ ./install.sh --install-python3 --venv /opt/cycle/gridengine/venv
ERROR: Package 'requests' requires a different Python: 3.6.8 not in '<4,>=3.7'
---- End output of "bash" "/tmp/chef-script20220913-26255-1e86596" ----
Ran "bash" "/tmp/chef-script20220913-26255-1e86596" returned 1
[2022-09-13T17:49:54+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
```

My line 526 in scheduler.rb equates to line 488. I have added some logic higher up in the file to properly install epel-release for RHEL to fix the package install of `jemalloc`. The offending portion of the bash command is the install.sh. If you run that section at the command line you receive the same error:

```
./install.sh --install-python3 --venv $INSTALLDIR/venv
INSTALL_PYTHON3=1
INSTALL_VIRTUALENV=1
VENV=/opt/cycle/gridengine/venv
created virtual environment CPython3.6.8.final.0-64 in 1034ms
creator CPython3Posix(dest=/opt/cycle/gridengine/venv, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
added seed packages: pip==21.3.1, setuptools==59.6.0, wheel==0.37.1
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Processing ./packages/certifi-2022.6.15-py3-none-any.whl
Processing ./packages/charset_normalizer-2.1.1-py3-none-any.whl
Processing ./packages/cyclecloud_api-8.0.1-py2.py3-none-any.whl
Processing ./packages/cyclecloud-gridengine-2.0.13.tar.gz
Preparing metadata (setup.py) ... done
Processing ./packages/cyclecloud-scalelib-0.1.7.tar.gz
Preparing metadata (setup.py) ... done
Processing ./packages/idna-3.3-py3-none-any.whl
Processing ./packages/immutabledict-1.0.0-py3-none-any.whl
Processing ./packages/importlib_metadata-3.6.0-py3-none-any.whl
Processing ./packages/jsonpickle-1.4.1-py2.py3-none-any.whl
Processing ./packages/requests-2.28.1-py3-none-any.whl
Processing ./packages/six-1.16.0-py2.py3-none-any.whl
Processing ./packages/typing_extensions-4.3.0-py3-none-any.whl
Processing ./packages/urllib3-1.26.12-py2.py3-none-any.whl
Processing ./packages/zipp-3.6.0-py3-none-any.whl
INFO: pip is looking at multiple versions of importlib-metadata to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of idna to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of jsonpickle to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of immutabledict to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of cyclecloud-scalelib to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of cyclecloud-gridengine to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of cyclecloud-api to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of charset-normalizer to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of certifi to determine which version is compatible with other requirements. This could take a while.
ERROR: Package 'requests' requires a different Python: 3.6.8 not in '<4,>=3.7'
Traceback (most recent call last):
File "/opt/cycle/gridengine/venv/bin/azge", line 4, in
from gridengine.cli import main
ModuleNotFoundError: No module named 'gridengine'
```

I have opened requests-2.28.1-py3-none-any.whl inside the cyclecloud-gridengine-pkg-2.0.13.tar.gz and found that 2.28.1 does not support python 3.6. From the METADATA file:

```
Metadata-Version: 2.1
Name: requests
Version: 2.28.1
Summary: Python HTTP for Humans.
Home-page: https://requests.readthedocs.io
Author: Kenneth Reitz
Author-email: me@kennethreitz.org
License: Apache 2.0
Project-URL: Documentation, https://requests.readthedocs.io
Project-URL: Source, https://github.com/psf/requests
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Web Environment
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Natural Language :: English
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.7, <4
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: charset-normalizer (<3,>=2)
Requires-Dist: idna (<4,>=2.5)
Requires-Dist: urllib3 (<1.27,>=1.21.1)
Requires-Dist: certifi (>=2017.4.17)
Provides-Extra: security
Provides-Extra: socks
Requires-Dist: PySocks (!=1.5.7,>=1.5.6) ; extra == 'socks'
Provides-Extra: use_chardet_on_py3
Requires-Dist: chardet (<6,>=3.0.2) ; extra == 'use_chardet_on_py3'

----------

Requests officially supports Python 3.7+.

```

From the [requests github](https://github.com/psf/requests), python 3.6 support was dropped in [release 2.28.0](https://github.com/psf/requests/blob/4d394574f5555a8ddcc38f707e0c9f57f55d9a3b/HISTORY.md#2280-2022-06-09). Cyclecloud-gridengine 2.0.11 did not experience this problem as at the time the release package was built, requests was on version 2.26.0. When 2.0.13 was packaged as a fix for an autoscaling issue we experienced, requests version 2.28.1 was the latest available. I think one fix is to version pin the dependency for requests to <=2.27.1. Attached is a screen capture of the 2.0.11 package contents at left and the 2.0.13 contents at right.
![Cyclecloud-gridgengine 2 0 11vs2 0 13](https://user-images.githubusercontent.com/86731297/190013613-c9bb4064-e495-4c7e-8f9f-6d2171c61f6b.png)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.