[BUG] SFTP driver caps download throughput at ~1 MB/s on higher-latency links (reads are never pipelined; WriteTo concurrency is never used)

Please confirm the following

• I have read and agree to AGPL-3.0 Section 15 .
  The program is provided "as is" without any warranties; you bear all risks of using it.

• I have read and agree to AGPL-3.0 Section 16 .
  The copyright holders and distributors are not liable for any damages resulting from the use or inability to use the program.

• I confirm my description is clear, polite, helps developers quickly locate the issue, and complies with community rules.

• I have read the OpenList documentation.

• I confirm there are no duplicate issues or discussions.

• I confirm this is an OpenList issue, not caused by other reasons (such as network, dependencies, or operation).

• I believe this issue must be handled by OpenList and not by a third party.

• I confirm this issue is not fixed in the latest version.

• I have not read these checkboxes and therefore I just ticked them all, Please close this issue.

OpenList Version (required)

v4.2.6

Storage Driver Used (required)

sftp

Bug Description (required)

[BUG] SFTP driver caps download throughput at ~1 MB/s on higher-latency links (reads are never pipelined; WriteTo concurrency is never used)

Environment

• OpenList version: v4.2.6 (also present on current main, HEAD f18b4ac)
• Deployment:
• Host OS / arch:
• SFTP remote:
• Approx. RTT to the SFTP remote:
• pkg/sftp: v1.13.11 (from go.mod)

Summary

Downloading a file from an SFTP storage through OpenList's proxy is capped at roughly 1 MB/s, regardless of the available bandwidth on either side. The throughput scales inversely with network latency, which is the signature of a latency-bound, one-request-at-a-time transfer, not a bandwidth limit.

The same file, fetched from the same remote over the same link with a plain client (scp, or sftp with a larger buffer), saturates the link. So the bottleneck is in how OpenList's SFTP driver reads, not in the network or the server.

There is currently no storage/config option to tune this — the SFTP driver's settings only expose address, credentials, root path and ignore_symlink_error.

Steps to reproduce

1. Add an SFTP storage pointing to a remote with non-trivial RTT (e.g. a VPS ~50 ms away).
2. Download a large file (≥ 100 MB) through OpenList (proxied).
3. Observe throughput plateauing around ~1 MB/s.
4. For comparison, on the same OpenList host: sftp -B 262144 user@remote:/path/file . (or scp) — this saturates the link.

Rough check: measured throughput ≈ 64 KiB / RTT. At ~60 ms RTT that is ≈ 1 MB/s, which matches the observation.

Root cause

The whole-file read path is strictly sequential and uses small chunks, and it never engages pkg/sftp's concurrent-read engine:

1. drivers/sftp/util.go → _initClient() builds the client with sftp.NewClient(conn) and no options, so maxPacket stays at the default 32768 bytes (32 KiB).

2. drivers/sftp/driver.go → Link() wraps the opened *sftp.File in stream.GetRangeReaderFromMFile, which returns an io.NewSectionReader over the file (internal/stream/util.go). All reads therefore go through ReadAt, and (*sftp.File).WriteTo is never called.

3. internal/net/serve.go drains that reader with utils.CopyWithBufferN(w, sendContent, sendSize), using the shared IoBuffPool buffer of 32*1024*2 = 64 KiB (pkg/utils/io.go).

Consequence, per iteration: ReadAt(64 KiB). In pkg/sftp v1.13.11, readAt only splits a read into concurrent sub-requests when len(b) > maxPacket, so a 64 KiB buffer yields at most two 32 KiB FXP_READ in flight. The outer copy is fully sequential (read 64 KiB → write 64 KiB → read …), so there is no overlap between the SFTP fetch and the HTTP send, and no pipelining across successive reads. Effective throughput ≈ 64 KiB / RTT — latency-bound.

Meanwhile (*sftp.File).WriteTo would spawn up to MaxConcurrentRequestsPerFile workers (default 64), sized to the file, overlapping round-trips — but it is never reached because the read goes through SectionReader.ReadAt.

Proposed fix

Two localized changes (happy to open a PR):

1. Build the client with sensible options — drivers/sftp/util.go, in _initClient():

d.client, err = sftp.NewClient(conn,
    // Larger packets; each concurrent worker fetches up to maxPacket per request.
    // >32 KiB requires a server that supports it (OpenSSH does). Fall back to
    // 1<<15 if a server rejects it ("failed to send packet header: EOF").
    sftp.MaxPacketUnchecked(1 << 17), // 128 KiB
    sftp.MaxConcurrentRequestsPerFile(64),
)

2. Stream full-file downloads through WriteTo so the concurrent-read engine is actually used — drivers/sftp/driver.go, Link() (needs io and pkg/http_range imports):

func (d *SFTP) Link(ctx context.Context, file model.Obj, args model.LinkArgs) (*model.Link, error) {
	if err := d.clientReconnectOnConnectionError(); err != nil {
		return nil, err
	}
	remoteFile, err := d.client.Open(file.GetPath())
	if err != nil {
		return nil, err
	}
	size := file.GetSize()

	rangeReaderFunc := func(ctx context.Context, httpRange http_range.Range) (io.ReadCloser, error) {
		length := httpRange.Length
		if length < 0 || httpRange.Start+length > size {
			length = size - httpRange.Start
		}
		// "To EOF" without a server-side speed limit: use (*sftp.File).WriteTo,
		// which performs concurrent reads (up to MaxConcurrentRequestsPerFile
		// workers) and overlaps network round-trips.
		if httpRange.Start+length >= size && stream.ServerDownloadLimit == nil {
			if _, err := remoteFile.Seek(httpRange.Start, io.SeekStart); err != nil {
				return nil, err
			}
			pr, pw := io.Pipe()
			go func() {
				_, werr := remoteFile.WriteTo(pw)
				_ = pw.CloseWithError(werr)
			}()
			return utils.NewReadCloser(pr, func() error { return pr.Close() }), nil
		}
		// Partial range: keep the original sequential ReadAt behavior.
		mFile := &stream.RateLimitFile{
			File:    remoteFile,
			Limiter: stream.ServerDownloadLimit,
			Ctx:     ctx,
		}
		return &model.FileCloser{File: io.NewSectionReader(mFile, httpRange.Start, length)}, nil
	}

	return &model.Link{
		RangeReader:      stream.RateLimitRangeReaderFunc(rangeReaderFunc),
		SyncClosers:      utils.NewSyncClosers(remoteFile),
		RequireReference: true,
	}, nil
}

Caveats / notes

• Partial Range requests (resume, video seeking) fall back to the original sequential ReadAt path — WriteTo only covers "current offset → EOF". A typical full download (the case people benchmark) is fixed.
• MaxPacketUnchecked > 32 KiB assumes the server accepts larger packets. If some servers reject it, keep the default 32 KiB — most of the gain comes from WriteTo overlapping round-trips anyway.
• When a server-side download speed limit (ServerDownloadLimit) is configured, the WriteTo fast path is intentionally skipped (the limiter caps speed regardless).
• Concurrent reads are already enabled by default in pkg/sftp; the problem is purely that the current read path never routes through the code that uses them.

Logs (required)

Configuration File Content (required)

Reproduction Link (optional)

No response

AI Generated Content

• I used AI tools to generate this content
• I did not use AI tools to generate this content

AI model used

Claude opus 4.8

Read drivers/sftp/util.go at _initClient and drivers/sftp/driver.go at Link, then trace internal/stream/util.go and internal/net/serve.go to confirm the sequential read path. Reproduce the latency-bound download and verify that full-file transfers overlap SFTP reads while partial ranges and server-side limits retain their existing behavior.