OpenSSH is Incompatible with Docker for Windows

My detailed write-up and video reproduction showing this incompatibility should more than answer the template questions including the versions. As you will see this is a rather mysterious issue that is very easily reproduced involving specifically OpenSSH-Win32 and Docker. It has been plaguing Docker users now for years and nobody (that I can find) ever could put their finger on the cause. Yesterday I opened this same issue over at Docker so I will simply paste it here in hopes between Docker and OpenSSH folks, someone can look through the code a bit more to determine just what is happening here:

---

Yesterday I tweeted and posted a video of the root cause of this (and almost every other Docker-Machine on Windows) error I have encountered.

Below I have posted a recap of the primary issue I experienced recently again and frankly for YEARS. This has been frustrating when all these Github issues seem to continually be erroneously closed to leave us having to perform a vast range of attempted workarounds without ever determining and addressing the root cause.

I personally have wasted hours probably totaling into the hundreds now troubleshooting this and similar various Docker on Windows issues and in all that time have never seen this resolution posted. It is certainly possible I missed it and if I did please feel free to point me to a dated write-up showing this as the root cause and I will absolutely stand corrected but otherwise I do believe this is the first time the actual root cause has been fully demonstrated with a solution.

Also of note: Please read to the end of this information because a common response may be that my findings are based on recent releases yet as I post below I can demonstrate this root cause has been in place for YEARS (I tested all the way back to the initial release of Docker and the results are the same!).

In my three decades in the industry, uncovering this one still felt pretty significant but then again when you've been banging your head against the wall for years on something, it usually does. :)

Background

There have been Github issues opened and subsequently closed dating all the way back to ISSUE #66 - very soon after the initial release yet closed. Many of these appear to be reproducible under this issue's cause or in some way related to this issue:

• https://github.com/docker/toolbox/issues/66
• https://github.com/docker/machine/issues/4447
• https://github.com/docker/toolbox/issues/547
• https://github.com/docker/toolbox/issues/116
• https://github.com/docker/toolbox/issues/153
• https://github.com/docker/toolbox/issues/473
• https://github.com/docker/toolbox/issues/329
• https://github.com/docker/machine/issues/986
• https://github.com/docker/machine/issues/732

And the Docker forums:

• https://forums.docker.com/t/windows-unable-to-quickstart/12536
• https://forums.docker.com/t/windows-unable-to-start-docker/12563
• https://forums.docker.com/t/error-getting-ip-address-something-went-wrong-running-an-ssh-command/15413
• https://www.bountysource.com/issues/43180735-cannot-run-docker-from-windows
• https://www.bountysource.com/issues/27997258-error-creating-machine-error-in-driver-during-machine-creation-maximum-number-of-retries-5-exceeded
• https://www.bountysource.com/issues/27997258-error-creating-machine-error-in-driver-during-machine-creation-maximum-number-of-retries-5-exceeded

And let's not forget all of StackOverflow sites (I'm only posting a couple but there are MANY):

• https://stackoverflow.com/questions/40378725/docker-quickstart-terminal-exit-status-255
• https://superuser.com/questions/996785/docker-toolbox-error-creating-machine-error-in-driver-during-machine-creation

Also, this issue's root cause is also responsible for all these Kitematic issues including:

• https://github.com/docker/kitematic/issues/862
• https://github.com/docker/kitematic/issues/1029

And if that is not enough, just run these two Google searches below, and you can bet that most if not all those hits with these "mystery" 255/timeouts/nondescriptive are also this issue:

• https://www.google.com/search?q=%22Could%20not%20determine%20IP%20from%20docker-machine%22
• https://www.google.com/search?q=docker%2Dmachine%20ssh%20exit%20status%20255

Issue(s)

• Management of Docker using Docker-Machine on Windows is impossible using the native shells of Powershell and CMD under certain conditions.

• Management of Docker using Docker-Machine on any OS may fail with recurring SSH errors.

Cause

Incompatible SSH client implementation.

Tests were run from the current beta release of OpenSSH_for_Windows_7.6p1, LibreSSL 2.6.4 OpenSSH-Win32 all the way back to OpenSSH_7.1p1 Microsoft Pragma Win32 port Oct 7 2015, OpenSSL 1.0.2d 9 Jul 2015 all tested versions exhibited this behavior.

Something in the call to "ip addr show" and possibly other operations from Docker-Machine are being interpreted incorrectly resulting in a terminating error. This prevents any successful Docker operations using either native shell in Windows.

This is becoming a bigger and bigger issue now since the incompatible client is automatically installed with Powershell and Chocolatey and added to the system path. The presence and priority in the system path can be changed and in such is the reason this issue does not appear to plague everyone on Windows and has been so difficult to troubleshoot.

Because the Mingwin bash shell relies on the separately installed Git SSH client, the QuickStart Terminal (usually) works which also had added to the difficulty in troubleshooting. However, once you move to native shells to manage the containers created with the QuickStart terminal, the system path will quickly prioritize the problematic SSH client and cause failures.

Based on testing I would recommend until full resolution that this could possibly be utilized as an accurate statement that sums up in one paragraph the current status of this issue:

Due to current incompatibilities, it is impossible for a user running any version of OpenSSH-Win32, installed by default with Powershell & Chocolatey, to install any version of Docker-Toolbox and utilize the native two shells (Powershell and CMD) to manage Docker without first taking action to disable OpenSSH-Win32 as the system default SSH client.

Note that the real solution is (and I am also immediately opening a Github issue there as well) for the Powershell team now maintaining the OpenSSH for Windows project to work with the Docker team to capture the low-level debug output from OpenSSH back to Docker-Machine's call to "ip addr show" (and other commands) to determine what on earth is being interpreted as a bad return, despite as I demonstrated the command clearly returning successfully the exact same output as the versions using OpenSSL instead of LibreSSL, and the connection via TLS being successful in all versions.

Issue Reproduction

Workaround

Until the root cause of the incompatibility with the OpenSSH-Win32 client can be addressed, there is only one workaround that has been successfully tested.

Both the SYSTEM and USER environment variable for PATH must be edited and any references to OpenSSH-Win32 (Default path of C:\Program Files\OpenSSH-Win64) must be moved BELOW another compatible SSH client. For example, the Git installed version is compatible so if Git were installed the path of C:\Program Files\Git\usr\bin should be moved before the OpenSSH version.

After making this change, a reboot is recommended.

No repository file or test is named. Start by reproducing the failure with OpenSSH-Win32 through Docker-Machine's "ip addr show" call and capture the low-level SSH debug output; done means identifying the incompatibility and verifying a fix against the reported Windows client versions and native shells.