SftpClient: Hang after ~500 MB with Dropbear server

After downloading about 500 MB of data, I'm getting a hang in DownloadFile.

I tested with files of various sizes: with a 1 MB (1,048,576 bytes) test file, 475 downloads work, and it deadlocks on the 476th one. With 2 MB files, 237 work, and it locks on the 238th. In all cases, it deadlocks shortly before 499,000,000 bytes.

This only seems to be an issue with the Dropbear SSH server with OpenSSH's sftp. When using OpenSSH on a desktop Linux, this issue does not occur.

I am assuming that this is not (strictly speaking) a bug in SSH.Net. Since it works with OpenSSH, I'm assuming it's Dropbear not acting quite right. But supporting slightly misbehaving SSH servers is a reasonable thing for SSH.Net to do.

Here is my test program.

using Renci.SshNet;
using System.Diagnostics;

namespace SshTransferTest;

internal class Program
{
    // This is my small embedded device. Low power. Running PetaLinux on a Xilinx SoC.
    // SSH Server is Dropbear v2020.80. (No, I can't change that.)

    // (Why 127.0.0.1? I'm on a work computer, with a strict VPN. I have to use an SSH tunnel
    // to make connections to machines on the local network.)
    const string host = "127.0.0.1";
    const int port = 2122;
    const string user = "root";
    const string privKey = "DeviceKey.id_rsa";
    const string largeFile = "/mnt/data/testLargeFile"; // 1.0 MB of /dev/urandom

    static void Main(string[] args)
    {
        Stopwatch sw = new();
        void callback(object? _)
        {
            Console.WriteLine("Running for {0}", sw.Elapsed);
            if (sw.Elapsed > TimeSpan.FromSeconds(30))
                Debugger.Break();
        }
        Timer lockupTimer = new(callback);

        SftpClient sftpClient = new(
            host, port, user,
            new PrivateKeyFile(typeof(Program).Assembly.GetManifestResourceStream(typeof(Program), privKey)!));

        sftpClient.Connect();

        for (int i = 0; i < 1000000; i++)
        {
            Console.WriteLine("{0}: Starting Download", i);
            lockupTimer.Change(TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(5));

            sw.Restart();
            using (MemoryStream ms = new())
            {
                sftpClient.DownloadFile(largeFile, ms);
            }

            sw.Stop();
            lockupTimer.Change(Timeout.InfiniteTimeSpan, Timeout.InfiniteTimeSpan);
            Console.WriteLine("{0}: Complete in {1}", i, sw.Elapsed);

            // I tried having a sleep here, but it didn't seem to make any difference.
            //Thread.Sleep(1000);
        }
    }
}

When it deadlocks, here's what the call stack looks like:

• Renci.SshNet.dll!Renci.SshNet.Sftp.SftpFileReader.Read() Line 106
• Renci.SshNet.dll!Renci.SshNet.SftpClient.InternalDownloadFile(string path, System.IO.Stream output, Renci.SshNet.Sftp.SftpDownloadAsyncResult asyncResult, System.Action downloadCallback) Line 1371
• Renci.SshNet.dll!Renci.SshNet.SftpClient.DownloadFile(string path, System.IO.Stream output, System.Action downloadCallback) Line 801
• SshTransferTest.dll!SshTransferTest.Program.Main(string[] args) Line 62

Here's the exact line where it's deadlocks in SftpFileReader.Read():

public byte[] Read()
{
	ThrowHelper.ThrowObjectDisposedIf(_disposingOrDisposed, this);
	if (_exception != null)
	{
		ExceptionDispatchInfo.Capture(_exception).Throw();
	}
	if (_isEndOfFileRead)
	{
		throw new SshException("Attempting to read beyond the end of the file.");
	}
	BufferedRead value;
	lock (_readLock)
	{
		while (!_queue.TryGetValue(_nextChunkIndex, out value) && _exception == null)
		{
			Monitor.Wait(_readLock); //<-------------------
		}

Other things I tried

Disconnect/Reconnect

for (int i = 0; i < 1000000; i++)
{
    if (i % 200 == 0)
    {
        sftpClient.Disconnect();
        sftpClient.Connect();
    }

    Console.WriteLine("{0}: Starting Download", i);

With this change, the deadlock does not occur. (Tested about 120,000 iterations.) So it appears to be a limit on the number of bytes transferred per SFTP connection, not per SftpClient object lifetime.

Different Download Method

//using (MemoryStream ms = new())
//{
//    sftpClient.DownloadFile(largeFile, ms);
//}

using (var sftpStream = sftpClient.OpenRead(largeFile))
{
    byte[] temp = new byte[65536];
    long red;
    while ((red = sftpStream.Read(temp)) != 0)
    {
        // Keep reading
        lockupTimer.Change(TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(5));
    }
}

With this change, the issue is still present. It still deadlocks, and still on the 476th download. With a different array size, still a deadlock on the 476th download.

Beginne bei SftpClient.DownloadFile und InternalDownloadFile und untersuche anschließend SftpFileReader.Read an dem im Bericht beschriebenen Monitor.Wait-Aufruf. Reproduziere wiederholte Downloads mit Dropbear v2020.80 und vergleiche sie mit OpenSSH, einschließlich des OpenRead-Pfads. Als abgeschlossen gilt die Aufgabe, wenn wiederholte Übertragungen nicht mehr bei ungefähr 499.000.000 Bytes hängen bleiben und die bestehenden Aufrufpfade weiterhin normal abgeschlossen werden.