core.remote action doesn't always capture stdout and truncates stderr

After upgrade to 2.6.0 I ran into a weird issue with core.remote actions.

Here's the details of the "faulty" execution (I masked internal-only domain with example.com):

$ st2 execution get 5a6f9c1ebd00ef78ff5fec97 -d
+-----------------+--------------------------------------------------------------+
| Property        | Value                                                        |
+-----------------+--------------------------------------------------------------+
| id              | 5a6f9c1ebd00ef78ff5fec97                                     |
| action.ref      | core.remote                                                  |
| context.user    | st2admin                                                     |
| parameters      | {                                                            |
|                 |     "cmd": "curl  -svL -w '%{http_code}'                     |
|                 | https://nexus3.example.com; >&2 echo " - something added      |
|                 | manually"",                                                  |
|                 |     "hosts": "127.0.0.1"                                     |
|                 | }                                                            |
| status          | succeeded (2s elapsed)                                       |
| start_timestamp | Mon, 29 Jan 2018 16:11:42 CST                                |
| end_timestamp   | Mon, 29 Jan 2018 16:11:44 CST                                |
| result          | {                                                            |
|                 |     "127.0.0.1": {                                           |
|                 |         "failed": false,                                     |
|                 |         "stderr": "* About to connect() to nexus3.example.com |
|                 | port 443 (#0)                                                |
|                 | *   Trying 10.143.16.34... connected                         |
|                 | * Connected to nexus3.example.com (10.143.16.34) port 443     |
|                 | (#0)                                                         |
|                 | * Initializing NSS with certpath: sql:/etc/pki/nssdb         |
|                 | *   CAfile: /etc/pki/tls/certs/ca-bundle.crt                 |
|                 |   CApath: none                                               |
|                 | * SSL connection using TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA    |
|                 | * Server certificate:                                        |
|                 | * \tsubject: CN=*.example.com,OU=Domain Control Validated     |
|                 | * \tstart date: Nov 10 14:21:01 2017 GMT                     |
|                 | * \texpire date: Nov 10 14:21:01 2020 GMT                    |
|                 | * \tcommon name: *.example.com                                |
|                 | * \tissuer: CN=Go Daddy Secure Certificate Authority -       |
|                 | G2,OU=http://certs.godaddy.com/repository/,O="GoDaddy.com,   |
|                 | Inc.",L=Scottsdale,ST=Arizona,C=US                           |
|                 | > GET / HTTP/1.1                                             |
|                 | > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu)          |
|                 | libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18             |
|                 | libssh2/1.4.2                                                |
|                 | > Host: nexus3.example.com                                    |
|                 | > Accept: */*                                                |
|                 | > ",                                                         |
|                 |         "return_code": 0,                                    |
|                 |         "succeeded": true,                                   |
|                 |         "stdout": ""                                         |
|                 |     }                                                        |
|                 | }                                                            |
| liveaction      | {                                                            |
|                 |     "runner_info": {                                         |
|                 |         "hostname": "lvcops101.example.com",                  |
|                 |         "pid": 30775                                         |
|                 |     },                                                       |
|                 |     "parameters": {                                          |
|                 |         "cmd": "curl  -svL -w '%{http_code}'                 |
|                 | https://nexus3.example.com; >&2 echo " - something added      |
|                 | manually"",                                                  |
|                 |         "hosts": "127.0.0.1"                                 |
|                 |     },                                                       |
|                 |     "action_is_workflow": false,                             |
|                 |     "callback": {},                                          |
|                 |     "action": "core.remote",                                 |
|                 |     "id": "5a6f9c1ebd00ef78ff5fec96"                         |
|                 | }                                                            |
+-----------------+--------------------------------------------------------------+

Expected stdout with a http code and stderr ending with - something added manually, but it is not the case. This can happen 4-5 times in a row and then work as expected 4-5 times. So it is intermittent and reason is unknown. Something with paramiko ssh runner? If I remove -v from the flags to curl, it feels like everything's fine in 100% runs. Arekhi on Slack was able (sorta) to reproduce it. I am filing this issue so someone can take a look at it and possibly fix. We have some workflows that rely on http code from curl, they became unstable, because there's no stdout from time to time now. I removed -v to alleviate it, but I'd like to have some sort of http log recorded for audit and troubleshooting purposes.

Start by reproducing the core.remote execution with the shown verbose curl command and compare the captured stdout and stderr across repeated runs. Then inspect the Paramiko SSH runner's handling of remote command output; done means stdout consistently contains the HTTP code and stderr is not truncated, including when curl uses -v.