Hang on detached process

Prerequisites

• Write a descriptive title.
• Make sure you are able to repro it on the latest version
• Search the existing issues.

Steps to reproduce

This is a complicated issue and I'm not sure if there is any solution but I thought it best to bring to your attention just in case.

There is a chance for a non-interactive SSH session to wait indefinitely for the stdout/stderr named pipes to signal EOF even when the immediate child process has ended.

Typically sshd will launch the immediate process with the stdout/stderr entries being inheritable handles, so sshd can get the output, and within a job that has set JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE. As this immediate process has the only open job handle, duplicated into it after CreateProcess, when it exits Windows will kill any other process in that job. For example any child process, the conhost.exe associated with the process and so on. Typically this is enough to ensure that all client handles of the stdout/stderr named pipe have been released so that sshd sees the EOF and finish its work but there is a way for a process to break this logic.

Any process within the process tree launched by sshd can create a new process with the following options:

• bInheritHandles is true
  • All inheritables handles are inherited to child process
  • More importantly, this means the stdout/stderr handles are inherited to the new process
• dwCreationFlags has set CREATE_BREAKAWAY_FROM_JOB
  • Escapes being killed when the main process ends
• dwCreationFlags has set CREATE_NEW_CONSOLE (only if launching a console exe)
  • No longer gets killed when the main process' conhost.exe ends with the job kill

The new process now holds a handle on the stdout/stderr named piped due to them being inheritable. As the process will continue running beyond the exit of the main process, sshd will continue to wait for EOF on those pipes.

I wrote https://gist.github.com/jborean93/76074b1a1f89553ffb36f563c9824c5c to help debug this problem. when I run it from an SSH client like ssh user@windows-host pwsh -File C:\\remote\\test-ssh-hand.ps1 I get this output

This shows that the immediate child process has stdout set to the named pipe W32PosixPipe.00002234.00000006 with the HANDLE value of 0x218. The script then launches cmd with the options listed above, waits a few seconds just so I could grab some screenshots and exits.

Looking at the process tree I can verify that pwsh.exe has ended and all that is left from the session is sshd.exe launched for this ssh command. I can also see that the cmd.exe grandchild process that was launched outside of the job and with a new console inherited the stdout named pipe handle stopping sshd.exe from seeing the EOF and finishing its work

As for how to fix this I don't really know how. You could not allow the breakway from job flag but this lead to https://github.com/PowerShell/Win32-OpenSSH/issues/1032 and is a slight disconnect with how nix hosts work. Maybe there's a way to set a timeout on the pipe reads when we know the main process has ended so it at least exits eventually rather than waiting forever. We can't not set the pipes to be inheritable as that's needed to set them as the stdout/stderr handles for the child process sshd spawns.

Expected behavior

sshd exits when the main process job reaps everything.

Actual behavior

sshd hangs until either the stdio pipes are manually closed in the grandchild process or it ends through some other mechanism.

Interactive ssh is not affected because these stdio handles aren't created/inherited or waiting on.

Error details

No error, just the hang

Environment data

Not applicable

Version

9.5.0.1

Visuals

No response

Start by reproducing the non-interactive SSH case with the linked debugging gist and inspect how sshd waits for EOF on stdout/stderr named pipes after the immediate child exits. Trace the CreateProcess handle inheritance and job-object behavior; done means the sshd session exits instead of waiting indefinitely when a breakaway grandchild retains the pipes.