Timeout throws unhandled exception

Observed in releases 2024.1.0 and 2024.2.0

When running an SshCommand with a timeout there's a risk of getting an unhandled exception if the underlying connection is disconnected/disposed before the command completes. The command will hang until the timeout is triggered and when the the timeout is triggered an exception is raised which cannot be caught.

Timeouts are implemented using a cancellation token that executes a delegate command when triggered. The problem seems to be that the delegate does not catch any exceptions thrown when it attempts to cancel the SshCommand and exceptions are not propagated to the client application.

https://github.com/sshnet/SSH.NET/blob/29997aebe1fdcc21ff329ca982f4a14c2b90f2b7/src/Renci.SshNet/SshCommand.cs#L292-L304

Code where the exception happens:

 try
    {
        cmdObj = ssh.CreateCommand(cmd);
        cmdObj.CommandTimeout = TimeSpan.FromSeconds(timeout);
        cmdObj.Execute();
        int exitStatus = cmdObj.ExitStatus ?? -1;
        res = (cmdObj.Result, exitStatus, cmdObj.Error);
        cmdObj?.Dispose();
        success = true;
    }
    catch (Exception e)
    {
        Log.Error(e.ToString());
    }

The code works well under normal circumstances, but after we wrote an integration test that causes the connection (ssh) to be disconnected immediately after a command is executed, the problem was discovered.

The exception causes the application to terminate with the following stacktrace captured using an exception logger attached to "AppDomain.CurrentDomain.UnhandledException"

---> Renci.SshNet.Common.SshConnectionException: Client not connected.
at Renci.SshNet.Session.SendMessage(Message message)
at Renci.SshNet.Channels.ChannelSession.SendSignalRequest(String signalName)
at Renci.SshNet.SshCommand.CancelAsync(Boolean forceKill, Int32 millisecondsTimeout)
at Renci.SshNet.SshCommand.<>c.b__43_0(Object cmd)
at System.Threading.CancellationTokenSource.Invoke(Delegate d, Object state, CancellationTokenSource source)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
--- End of inner exception stack trace ---
at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
at System.Threading.TimerQueue.FireNextTimers()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
2024-12-24 02:22:44.990 +00:00 [FTL] Inner exception: Renci.SshNet.Common.SshConnectionException: Client not connected.
at Renci.SshNet.Session.SendMessage(Message message)
at Renci.SshNet.Channels.ChannelSession.SendSignalRequest(String signalName)
at Renci.SshNet.SshCommand.CancelAsync(Boolean forceKill, Int32 millisecondsTimeout)
at Renci.SshNet.SshCommand.<>c.b__43_0(Object cmd)
at System.Threading.CancellationTokenSource.Invoke(Delegate d, Object state, CancellationTokenSource source)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)

A workaround has been implemented by wrapping the command execution in a separate task and using a Task.Delay(timeout) to implement the timeout. Using the built-in SshCommand.CommandTimeout causes the unhandled exception to be raised.

Start in src/Renci.SshNet/SshCommand.cs at the linked lines around CommandTimeout, then trace ExecuteAsync and CancelAsync in the stack trace. Reproduce the disconnected-connection case described in the issue and add or update an integration test for it. Done means a timeout after disconnection does not surface as an unhandled exception.