Azure / Azure/cyclecloud-gridengine

Cannot reflect proxy setting to run /install.sh --install-python3 --venv /opt/cycle/gridengine/venv in "cyclecloud-gridengine-pkg-2.0.12.tar.gz"

Open
#30 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7
Forks
18
Avg merge
5d 21h
Merged PRs (30d)
2

Description

I am deploying the GridEngine cluster (project version 2.0.12) under proxy-enabled environment. First, I added proxy setting in cloud-init as below and deployed, but it failed with the message below.

----------------------------------------------------------------------------------------
:
+ ./install.sh --install-python3 --venv /opt/cycle/gridengine/venv
which: no python3 in (/sched/sge/sge-2011.11/bin/linux-x64:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin)
WARNING: Running pip install with root privileges is generally not a good idea. Try `__main__.py install --user` instead.
Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable',)': /simple/virtualenv/
:
Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable',)': /simple/virtualenv/
Could not find a version that satisfies the requirement virtualenv (from versions: )
No matching distribution found for virtualenv
EXCEPTION: bash[setup cyclecloud-gridengine] (gridengine::scheduler line 488) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
----------------------------------------------------------------------------------------

Therefore, as a workaround to avoid the situation above, I did the following (1) to (5) instead.

(1) I added proxy setting in Cloud-init tab for the cluster setting on cyclecloud portal.
----------------------------------------------------------------------------------------
#!/bin/bash
export https_proxy="http://10.0.4.4:3128/"
export HTTPS_PROXY="http://10.0.4.4:3128/"
export http_proxy="http://10.0.4.4:3128/"
export HTTP_PROXY="http://10.0.4.4:3128/"
export NO_PROXY="127.0.0.1,localhost,.core.windows.net,10.0.0.0/16,10.0.1.0/24,10.0.2.0/24,10.0.3.0/24,10.0.4.0/24"
export no_proxy="127.0.0.1,localhost,.core.windows.net,10.0.0.0/16,10.0.1.0/24,10.0.2.0/24,10.0.3.0/24,10.0.4.0/24"
cat <> /etc/environment
https_proxy="http://10.0.4.4:3128/"
HTTPS_PROXY="http://10.0.4.4:3128/"
http_proxy="http://10.0.4.4:3128/"
HTTP_PROXY="http://10.0.4.4:3128/"
NO_PROXY="127.0.0.1,localhost,.core.windows.net,10.0.0.0/16,10.0.1.0/24,10.0.2.0/24,10.0.3.0/24,10.0.4.0/24"
no_proxy="127.0.0.1,localhost,.core.windows.net,10.0.0.0/16,10.0.1.0/24,10.0.2.0/24,10.0.3.0/24,10.0.4.0/24"
EOF
echo "proxy=http://10.0.4.4:3128" >> /etc/yum.conf
cat <> /etc/wgetrc
http_proxy=http://10.0.4.4:3128/
https_proxy=http://10.0.4.4:3128/
EOF2
----------------------------------------------------------------------------------------

(2) Next, I manually added proxy setting to "install.sh" in "cyclecloud-gridengine-pkg-2.0.12.tar.gz" on cyclecloud server.

----------------------------------------------------------------------------------------
#!/usr/bin/env bash

export https_proxy="http://10.0.4.4:3128/"
export HTTPS_PROXY="http://10.0.4.4:3128/"
export http_proxy="http://10.0.4.4:3128/"
export HTTP_PROXY="http://10.0.4.4:3128/"
export NO_PROXY="127.0.0.1,localhost,.core.windows.net,10.0.0.0/16,10.0.1.0/24,10.0.2.0/24,10.0.3.0/24,10.0.4.0/24"
export no_proxy="127.0.0.1,localhost,.core.windows.net,10.0.0.0/16,10.0.1.0/24,10.0.2.0/24,10.0.3.0/24,10.0.4.0/24"

INSTALL_PYTHON3=0
INSTALL_VIRTUALENV=0
INSTALL_PIP=0
INSTALLDIR=/opt/cycle/gridengine
VENV=$INSTALLDIR/venv
OS=$(awk -F= '/^NAME/{print $2}' /etc/os-release | awk '{print tolower($0)}')
:
----------------------------------------------------------------------------------------

(3) Next, I added following line in the template file.

----------------------------------------------------------------------------------------
:
[[[cluster-init cyclecloud/gridengine:default]]]
[[[cluster-init gridengine:default:2.0.12]]]
:
[[[cluster-init cyclecloud/gridengine:scheduler]]]
[[[cluster-init gridengine:scheduler:2.0.12]]]
:
[[[cluster-init cyclecloud/gridengine:execute]]]
[[[cluster-init gridengine:execute:2.0.12]]]
:
----------------------------------------------------------------------------------------

(4) Next, I ran the commands below to register the project to CycleCloud server.

$ cyclecloud delete_template ge-proxytest01
$ cyclecloud project upload azureuser-storage
$ cyclecloud import_template -f templates/gridengine.txt

(5) I deployed the cluster from cyclecloud portal. Finally, the cluster was deployed without python error.

However, I really feel that this is not a good way because I manually modified "install.sh" in "cyclecloud-gridengine-pkg-2.0.12.tar.gz" to add proxy settings. However, as "install.sh" is archived in ""cyclecloud-gridengine-pkg-2.0.12.tar.gz", this is the only way that I managed to solve the situation.

Are there any good way to reflect proxy setting without changing "install.sh" in "cyclecloud-gridengine-pkg-2.0.12.tar.gz"? under proxy-enabled environment? I would appreciate for the feedback.

Thanks.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with install.sh in cyclecloud-gridengine-pkg-2.0.12.tar.gz and the gridengine.txt template, then trace how the scheduler setup invokes ./install.sh --install-python3 --venv. Confirm how proxy settings from cloud-init reach that process. Done means the package installation can use the configured proxy without manually editing the archived install.sh.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash, python
Domain
cloud, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.