core.remote action throwing an error when ssh connection using private key

SUMMARY

When running core.remote action to ssh to host using private key error is thrown

STACKSTORM VERSION

st2 --version
st2 3.7.0, on Python 3.8.10

OS, environment, install method

K8s helm chart installation

Steps to reproduce the problem

st2 run core.remote cmd=whoami hosts=xxx username=stanley private_key=/home/stanley/.ssh/stanley_rsa

Expected Results

This action should run whoami command on remote host and print output.

SSH connection using this private_key is working properly password less when running ssh command from action_runner pod:

ssh -i stanley_rsa stanley@xxx

Exactly the same private_key we have on our old stackstorm instance(st2 3.5dev (596c60c23), on Python 3.6.9) and same core.remote action is able to run successfully. I have also verified that paramiko library has different versions:

• old stackstorm instance: paramiko-2.7.2.dist-info
• new stackstorm instance: paramiko-2.10.1.dist-info

Actual Results

st2 run core.remote cmd=whoami hosts=xxx username=stanley private_key=/home/stanley/.ssh/stanley_rsa

.

id: 63f779e3308e12af9365df26

action.ref: core.remote

context.user: st2admin

parameters: 

 cmd: whoami

 hosts: xxx

 private_key: '********'

 username: stanley

status: failed

start_timestamp: Thu, 23 Feb 2023 14:36:19 UTC

end_timestamp: Thu, 23 Feb 2023 14:36:21 UTC

result: 

 error: "Unable to connect to any one of the hosts: ['xxx'].

connect_errors={

 "xxx": {

   "failed": true,

   "succeeded": false,

   "timeout": false,

   "return_code": 255,

   "stdout": "",

   "stderr": "",

   "error": "Failed connecting to host xxx. q must be exactly 160, 224, or 256 bits long",

   "traceback": "Traceback (most recent call last):\

 File \\"/opt/stackstorm/st2/lib/python3.8/site-packages/st2common/runners/parallel_ssh.py\\", line 278, in _connect\

   client.connect()\

 File \\"/opt/stackstorm/st2/lib/python3.8/site-packages/st2common/runners/paramiko_ssh.py\\", line 171, in connect\

   self.client = self._connect(host=self.hostname, socket=self.bastion_socket)\

 File \\"/opt/stackstorm/st2/lib/python3.8/site-packages/st2common/runners/paramiko_ssh.py\\", line 787, in _connect\

   client.connect(**conninfo)\

 File \\"/opt/stackstorm/st2/lib/python3.8/site-packages/paramiko/client.py\\", line 435, in connect\

   self._auth(\

 File \\"/opt/stackstorm/st2/lib/python3.8/site-packages/paramiko/client.py\\", line 682, in _auth\

   self._transport.auth_publickey(username, key)\

 File \\"/opt/stackstorm/st2/lib/python3.8/site-packages/paramiko/transport.py\\", line 1634, in auth_publickey\

   return self.auth_handler.wait_for_response(my_event)\

 File \\"/opt/stackstorm/st2/lib/python3.8/site-packages/paramiko/auth_handler.py\\", line 244, in wait_for_response\

   raise e\

 File \\"/opt/stackstorm/st2/lib/python3.8/site-packages/paramiko/transport.py\\", line 2163, in run\

   handler(self.auth_handler, m)\

 File \\"/opt/stackstorm/st2/lib/python3.8/site-packages/paramiko/auth_handler.py\\", line 375, in _parse_service_accept\

   sig = self.private_key.sign_ssh_data(blob, algorithm)\

 File \\"/opt/stackstorm/st2/lib/python3.8/site-packages/paramiko/dsskey.py\\", line 109, in sign_ssh_data\

   key = dsa.DSAPrivateNumbers(\

 File \\"/opt/stackstorm/st2/lib/python3.8/site-packages/cryptography/hazmat/primitives/asymmetric/dsa.py\\", line 244, in private_key\

   return backend.load_dsa_private_numbers(self)\

 File \\"/opt/stackstorm/st2/lib/python3.8/site-packages/cryptography/hazmat/backends/openssl/backend.py\\", line 826, in load_dsa_private_numbers\

   dsa._check_dsa_private_numbers(numbers)\

 File \\"/opt/stackstorm/st2/lib/python3.8/site-packages/cryptography/hazmat/primitives/asymmetric/dsa.py\\", line 282, in _check_dsa_private_numbers\

   _check_dsa_parameters(parameters)\

 File \\"/opt/stackstorm/st2/lib/python3.8/site-packages/cryptography/hazmat/primitives/asymmetric/dsa.py\\", line 274, in _check_dsa_parameters\

   raise ValueError(\\"q must be exactly 160, 224, or 256 bits long\\")\

ValueError: q must be exactly 160, 224, or 256 bits long\

"

 }

}"

 traceback: "  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2actions/container/base.py", line 117, in _do_run

   runner.pre_run()

 File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2common/runners/paramiko_ssh_runner.py", line 206, in pre_run

   self._parallel_ssh_client = ParallelSSHClient(**client_kwargs)

 File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2common/runners/parallel_ssh.py", line 90, in __init__

   connect_results = self.connect(raise_on_any_error=raise_on_any_error)

 File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2common/runners/parallel_ssh.py", line 131, in connect

   raise NoHostsConnectedToException(msg)

"

Any suggestion what needs to be changed and how to make core.remote action work correctly?

Start with st2common/runners/paramiko_ssh.py and the connection flow through st2common/runners/parallel_ssh.py, focusing on the DSA private-key failure shown in the traceback. Reproduce core.remote with the reported private key and current Paramiko dependency, then verify that the action runs whoami successfully or reports a clear compatibility error.