No SSH connection possible "Error connecting to web socket"

Stratos Version

4.4.0

Frontend Deployment type

• [ x ] Cloud Foundry Application (cf push)
• Kubernetes, using a helm chart
• Docker, single container deploying all components
• npm run start
• Other (please specify below)

Backend (Jet Stream) Deployment type

• [ x ] Cloud Foundry Application (cf push)
• Kubernetes, using a helm chart
• Docker, single container deploying all components
• Other (please specify below)

Expected behaviour

SSH access to apps is possible. Stratos uses the correct id.

Actual behaviour

For several apps we can't create a ssh connection (via instances -> SSH) with stratos. We only get the error message "Error connecting to web socket".

After some debugging we discovered that Stratos is requesting with a "wrong" application id. Here are the logentries from the scheduler/ssh-proxy service in cloud foundry:

{​​​​​​​"timestamp":"2021-02-09T10:06:34.516532713Z","level":"error","source":"ssh-proxy","message":"ssh-proxy.authentication-failed","data":{​​​​​​​"error":"ssh: no auth passed yet","user":"cf:94c2a04d-eabc-44f9-a31f-0e9d666e9aec/0@ssh.mycompany"}​​​​​​​}​​​​​​​
{​​​​​​​"timestamp":"2021-02-09T10:06:34.517828483Z","level":"info","source":"ssh-proxy","message":"ssh-proxy.cf-authenticate.authenticate-starting","data":{​​​​​​​"session":"264"}​​​​​​​}​​​​​​​
{​​​​​​​"timestamp":"2021-02-09T10:06:34.858569794Z","level":"error","source":"ssh-proxy","message":"ssh-proxy.cf-authenticate.fetching-app-failed","data":{​​​​​​​"ResponseBody":{​​​​​​​}​​​​​​​,"StatusCode":"404 Not Found","app":"94c2a04d-eabc-44f9-a31f-0e9d666e9aec/0","error":"Fetching application data failed","principal":"99d98a36-8c29-416d-a424-f19289409848","session":"264","username":"me@mycompany"}​​​​​​​}​​​​​​​
{​​​​​​​"timestamp":"2021-02-09T10:06:34.858785507Z","level":"info","source":"ssh-proxy","message":"ssh-proxy.cf-authenticate.authenticate-finished","data":{​​​​​​​"session":"264"}​​​​​​​}​​​​​​​
{​​​​​​​"timestamp":"2021-02-09T10:06:34.858832877Z","level":"error","source":"ssh-proxy","message":"ssh-proxy.authentication-failed","data":{​​​​​​​"error":"Fetching application data failed","user":"cf:94c2a04d-eabc-44f9-a31f-0e9d666e9aec/0@ssh.mycompany"}​​​​​​​}​​​​​​​

So the app id "94c2a04d-eabc-44f9-a31f-0e9d666e9aec/0" could not be found.

Running the "cf cli" command for the same application results in

{"timestamp":"2021-02-09T10:26:08.465319920Z","level":"error","source":"ssh-proxy","message":"ssh-proxy.authentication-failed","data":{​​​​​​​"error":"ssh: no auth passed yet","user":"cf:4e874674-df2b-4eb1-ab62-ca90a3f4db91/0"}​​​​​​​}​​​​​​​
{​​​​​​​"timestamp":"2021-02-09T10:26:08.467275889Z","level":"info","source":"ssh-proxy","message":"ssh-proxy.cf-authenticate.authenticate-starting","data":{​​​​​​​"session":"738"}​​​​​​​}​​​​​​​
{​​​​​​​"timestamp":"2021-02-09T10:26:08.751682714Z","level":"info","source":"ssh-proxy","message":"ssh-proxy.cf-authenticate.app-access-success","data":{​​​​​​​"app":"4e874674-df2b-4eb1-ab62-ca90a3f4db91/0","principal":"99d98a36-8c29-416d-a424-f19289409848","session":"738","username":"me@mycompany"}​​​​​​​}​​​​​​​
{​​​​​​​"timestamp":"2021-02-09T10:26:08.751719507Z","level":"info","source":"ssh-proxy","message":"ssh-proxy.cf-authenticate.authenticate-finished","data":{​​​​​​​"session":"738"}​​​​​​​}​​​​​​​
{​​​​​​​"timestamp":"2021-02-09T10:26:08.751752850Z","level":"info","source":"ssh-proxy","message":"ssh-proxy.authentication-attempted","data":{​​​​​​​"user":"cf:4e874674-df2b-4eb1-ab62-ca90a3f4db91/0"}​​​​​​​}​​​​​​​
{​​​​​​​"timestamp":"2021-02-09T10:26:08.767173858Z","level":"info","source":"ssh-proxy","message":"ssh-proxy.handle-connection.new-client-conn.connected-to-backend","data":{​​​​​​​"backend-address":"10.176.213.8:61075","session":"737.1"}​​​​​​​}​​​​​​​
{​​​​​​​"timestamp":"2021-02-09T10:26:08.773429884Z","level":"info","source":"ssh-proxy","message":"ssh-proxy.handle-connection.from-client.proxy-global-requests.started","data":{​​​​​​​"session":"737.2.1"}​​​​​​​}​​​​​​​

and I can ssh into the app.

Seems like the cf cli (version 7.1.0) is using another id.

Having a closer look into the application we can see that cf cli is using the "process id" (4e874674-df2b-4eb1-ab62-ca90a3f4db91) and stratos the app id (94c2a04d-eabc-44f9-a31f-0e9d666e9aec).

VCAP_APPLICATION={"cf_api":"https://api.mycompany","limits":{"fds":16384,"mem":384,"disk":1024},"application_name":"myapp","application_uris":["myapp.mycompany"],"name":"myapp","space_name":"myspace","space_id":"d64180a4-e607-419f-92f9-d95c96ea7759","organization_id":"8611e8b0-ff63-4b76-9017-df69e8166331","organization_name":"myorg","uris":["myapp.mycompany"],"process_id":"4e874674-df2b-4eb1-ab62-ca90a3f4db91","process_type":"web","application_id":"94c2a04d-eabc-44f9-a31f-0e9d666e9aec","version":"43abc7e3-f40a-43b6-b8c4-9a88f0cd4ebd","application_version":"43abc7e3-f40a-43b6-b8c4-9a88f0cd4ebd"}

Maybe the difference is because the app was deployed with the CF 7 rolling app deployment (cf push APP-NAME --strategy rolling)?

Other applications (which we can access with stratos and cf cli) don't have different ids as app id and process id.

Start with the Stratos SSH connection flow and compare the Cloud Foundry scheduler/ssh-proxy logs for the Stratos request and the cf CLI request. Verify how application_id and process_id from VCAP_APPLICATION are selected; done means SSH succeeds for apps whose IDs differ, including apps deployed with the rolling strategy.