Old version of pip is installed into pack's virtualenv

SUMMARY

It seems that when the virtualenv's are built that some weird version of pip gets installed and I'm not sure where this is coming from. I even upgraded the system pip version to latest with no avail.

Because such an old version of pip gets installed it doesn't have support for some of the libs that get installed for things like st2client. Specifically orjson as it tries to build from source which requires rust to be installed.

STACKSTORM VERSION

st2 3.4.1, on Python 3.6.8

OS, environment, install method

HA cluster with two nodes and a controller.

Steps to reproduce the problem

Install a pack either manually or automatically. I'm not sure what the prereq's are. I have python 3.6 installed via yum. This installs an even older version of pip than what gets built in the virutalenv.

Expected Results

That the pip version from /opt/stackstorm/st2 gets installed and not some other version.

Actual Results

The install

<redacted> virtualenvs # /opt/stackstorm/st2/bin/virtualenv -p /opt/stackstorm/st2/bin/python --system-site-packages --verbose --no-download /opt/stackstorm/virtualenvs/zabbix -vvvv
Already using interpreter /opt/stackstorm/st2/bin/python
Using real prefix '/usr'
Creating /opt/stackstorm/virtualenvs/zabbix/lib/python3.6
This system uses lib64; symlinking lib64 to lib
Symlinking Python bootstrap modules
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib/python3.6/lib-dynload
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib/python3.6/config-3.6m-x86_64-linux-gnu
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/os.py
  Ignoring built-in bootstrap module: posix
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/posixpath.py
  Cannot import bootstrap module: nt
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/ntpath.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/genericpath.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/fnmatch.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/locale.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/encodings
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/codecs.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/stat.py
  Cannot import bootstrap module: UserDict
  File /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/lib-dynload/readline.cpython-36m-x86_64-linux-gnu.so already exists
  Cannot import bootstrap module: copy_reg
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/types.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/re.py
  Cannot import bootstrap module: sre
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/sre_parse.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/sre_constants.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/sre_compile.py
  File /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so already exists
  Cannot import bootstrap module: _abcoll
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/warnings.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/linecache.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/abc.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/io.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/_weakrefset.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/copyreg.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/tempfile.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/random.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/__future__.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/collections
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/keyword.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/tarfile.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/shutil.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/struct.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/copy.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/tokenize.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/token.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/functools.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/heapq.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/bisect.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/weakref.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/reprlib.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/base64.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/_dummy_thread.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/hashlib.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/hmac.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/imp.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/importlib
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/rlcompleter.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/operator.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/_collections_abc.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/_bootlocale.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/enum.py
  No LICENSE.txt / LICENSE found in source
Creating /opt/stackstorm/virtualenvs/zabbix/lib/python3.6/site-packages
Writing /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/site.py
Writing /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/orig-prefix.txt
Creating parent directories for /opt/stackstorm/virtualenvs/zabbix/include
Symlinking /opt/stackstorm/virtualenvs/zabbix/include/python3.6m
File /opt/stackstorm/virtualenvs/zabbix/lib/python3.6/lib-dynload already exists
File /opt/stackstorm/virtualenvs/zabbix/lib/python3.6/config-3.6m-x86_64-linux-gnu already exists
Creating /opt/stackstorm/virtualenvs/zabbix/bin
New python executable in /opt/stackstorm/virtualenvs/zabbix/bin/python
Changed mode of /opt/stackstorm/virtualenvs/zabbix/bin/python to 0o755
Testing executable with /opt/stackstorm/virtualenvs/zabbix/bin/python -c "import sys;out=sys.stdout;getattr(out, "buffer", out).write(sys.prefix.encode("utf-8"))"
Got sys.prefix result: '/st2content/virtualenvs/zabbix'
Creating /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/distutils
Writing /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/distutils/__init__.py
Writing /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/distutils/distutils.cfg
Running command /opt/stackstorm/virtualenvs/zabbix/bin/python -m pip config list
Installing setuptools, pip, wheel...

  Running command /opt/stackstorm/virtualenvs/zabbix/bin/python - setuptools pip wheel
  Looking in links: /opt/stackstorm/st2/lib/python3.6/site-packages/virtualenv_support
  Collecting setuptools
  Collecting pip
  Collecting wheel
  Installing collected packages: setuptools, pip, wheel
  Successfully installed pip-19.1.1 setuptools-41.0.1 wheel-0.33.4
...Installing setuptools, pip, wheel...done.
Writing /opt/stackstorm/virtualenvs/zabbix/bin/activate
Writing /opt/stackstorm/virtualenvs/zabbix/bin/activate.fish
Writing /opt/stackstorm/virtualenvs/zabbix/bin/activate.csh
Writing /opt/stackstorm/virtualenvs/zabbix/bin/activate.ps1
Writing /opt/stackstorm/virtualenvs/zabbix/bin/activate_this.py
Writing /opt/stackstorm/virtualenvs/zabbix/bin/activate.xsh
Writing /opt/stackstorm/virtualenvs/zabbix/bin/python-config
Changed mode of /opt/stackstorm/virtualenvs/zabbix/bin/python-config to 0o755

This is the installed pip versions. I also upgraded the system pip3 to version 21.2.2 and still get 19.1.1 installed.

(st2) <redacted> virtualenvs # for i in $(ls); do $i/bin/pip --version; done
pip 19.1.1 from /opt/stackstorm/virtualenvs/ansible/lib/python3.6/site-packages/pip (python 3.6)
pip 19.1.1 from /opt/stackstorm/virtualenvs/bitbucket/lib/python3.6/site-packages/pip (python 3.6)
pip 19.1.1 from /opt/stackstorm/virtualenvs/chef/lib/python3.6/site-packages/pip (python 3.6)
pip 19.1.1 from /opt/stackstorm/virtualenvs/docker/lib/python3.6/site-packages/pip (python 3.6)
pip 19.1.1 from /opt/stackstorm/virtualenvs/jenkins/lib/python3.6/site-packages/pip (python 3.6)
pip 19.1.1 from /opt/stackstorm/virtualenvs/msexchange/lib/python3.6/site-packages/pip (python 3.6)
pip 19.1.1 from /opt/stackstorm/virtualenvs/opsgenie/lib/python3.6/site-packages/pip (python 3.6)
pip 19.1.1 from /opt/stackstorm/virtualenvs/servicenow/lib/python3.6/site-packages/pip (python 3.6)
pip 19.1.1 from /opt/stackstorm/virtualenvs/splunk/lib/python3.6/site-packages/pip (python 3.6)
pip 19.1.1 from /opt/stackstorm/virtualenvs/zabbix/lib/python3.6/site-packages/pip (python 3.6)
pip 19.1.1 from /opt/stackstorm/virtualenvs/zabbix/lib/python3.6/site-packages/pip (python 3.6)
(st2) <redacted> virtualenvs # pip3 --version
pip 20.0.2 from /opt/stackstorm/st2/lib/python3.6/site-packages/pip (python 3.6)
(st2) <redacted> virtualenvs # /usr/bin/pip3 --version
pip 9.0.3 from /usr/lib/python3.6/site-packages (python 3.6)

Thanks!

Reproduce a pack installation using /opt/stackstorm/st2/bin/virtualenv with the shown Python 3.6 and --no-download options. Inspect the virtualenv bootstrap packages under /opt/stackstorm/st2/lib/python3.6/site-packages/virtualenv_support and compare them with the installed pip versions. Done means newly created pack virtualenvs use the expected pip version rather than 19.1.1 and dependencies such as orjson install without the reported failure.