apache / apache/opendal

sftp: Port the SFTP transport from openssh to russh

Open
#7,968 7 comments 2 reactions 0 assignees View on GitHub
enhancement research services/sftp
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

### Summary

Port the SFTP service from the `openssh` crate to the pure-Rust [`russh`](https://github.com/Eugeny/russh) + [`russh-sftp`](https://github.com/AspectUnk/russh-sftp) stack.

The current transport drives the system `ssh` binary through the `openssh` crate, which requires a local OpenSSH client and a Unix control socket. This makes the service Unix-only and dependent on an external binary. `russh` is a pure-Rust, Tokio-native SSH implementation that removes both constraints and unblocks Windows.

This proposes a **single, clean replacement** — not a dual backend. The goal is to keep the public SFTP API (`sftp://` scheme, `SftpConfig`, `Operator` behavior) unchanged while swapping the transport underneath.

### Motivation

- **No Windows support.** `openssh` needs a local `ssh` binary and a Unix control socket, so the service cannot build or run on Windows. This is the long-standing ask in #2963, and `openssh` has concretely failed to build on Windows in CI (#6773).
- **External runtime dependency.** The transport requires an `ssh` binary on `PATH` and filesystem access for the control socket, which complicates minimal/container images and sandboxed environments.
- **Out-of-process behavior.** Auth, known-hosts, and config resolution are partly delegated to the external `ssh` process rather than controlled in-process.

### Why russh

`russh` (v0.62.x, Apache-2.0, actively maintained, widely adopted) is a pure-Rust SSH2 client/server. `russh-sftp` (v2.3.x, Apache-2.0) provides the SFTP subsystem and works over any russh channel. `russh-sftp` targets SFTP v3, matching the current behavior.

**Gains**

- Cross-platform, including Windows (and wasm) — no system `ssh` binary, no control socket.
- Fully in-process SSH: OpenDAL controls auth, key handling, and host-key checks.
- Broad auth support out of the box: publickey, password, keyboard-interactive, OpenSSH certificates, and SSH agent (including Windows Pageant and named-pipe agents). Password login (#2966) becomes straightforward.
- All operations the service uses today are covered: open/read/seek (range reads), create/append/truncate, `if_not_exists`, list, delete, rename, canonicalize, stat, create_dir.

**Losses / costs**

- **known_hosts handling is not built-in.** `russh` exposes a host-key verification callback; the `strict` / `accept` / `add` semantics and `~/.ssh/known_hosts` read/write must be implemented in OpenDAL. Today `openssh` provides this for free. **This is the primary engineering effort of the port.**
- **No server-side copy in `russh-sftp`.** However, the current service's `copy` is already a client-side stream copy (open source, create destination, stream through the client), so this is reproduced with no capability loss — it just needs a manual reimplementation.
- Potentially larger dependency and binary footprint

Non-issues, explicitly out of scope:

- The mandatory `russh` crypto backend (`aws-lc-rs` / `ring`) is already present in OpenDAL's dependency tree via other services' TLS stacks, so it adds no new dependency category.
- `ProxyCommand` / `ProxyJump` / `ssh_config` resolution are not exposed by the current SFTP service, so there is nothing to regress; they are out of scope for this port.

### Acceptance criteria

- [ ] SFTP transport uses `russh` + `russh-sftp`; the `openssh` and `openssh-sftp-client` dependencies are removed.
- [ ] Public API is unchanged: `sftp://` scheme, `SftpConfig` fields (`endpoint`, `root`, `user`, `key`, `known_hosts_strategy`), and `Operator` behavior. New optional fields (e.g. `password`) may be added but existing ones keep their meaning.
- [ ] **known_hosts parity:** `strict` / `accept` / `add` strategies behave equivalently to the current `openssh`-backed implementation, including reading and updating `~/.ssh/known_hosts`. *(Primary effort.)*
- [ ] Auth parity for publickey (keyfile from `SftpConfig.key`) and known-hosts; password support wired in (addresses #2966).
- [ ] Capability parity: stat, read (with range), write (multi + `if_not_exists`), create_dir, delete, list (with limit), copy, rename, shared — verified against behavior tests.
- [ ] `copy` reimplemented as a client-side stream copy (matching current semantics).
- [ ] Connection pooling preserved (the `fastpool`-based pool and its recyclability check).
- [ ] Error mapping preserved: `NotFound` / `PermissionDenied` / `Unsupported` / `ConditionNotMatch` classifications re-implemented against `russh` / `russh-sftp` error types.
- [ ] Service builds and passes behavior tests on **Linux and Windows** in CI.
- [ ] Docs updated: remove the "only works on unix" note; document any auth/known-hosts differences.

### Impact

No intended change to the public SFTP API; the port swaps the transport and adds Windows support. Removes the "only works on unix" limitation and would resolve #2963.

**Cross-binding cleanup:** the Java, Python, and .NET bindings currently gate `services-sftp` off Windows because of `openssh`. Those gates can be removed as follow-ups once this lands (note: the .NET gate must keep the unrelated `services-monoiofs`).

### References

- Current service: `core/services/sftp/`
- Windows support request: #2963
- `openssh` failed to build on Windows in CI: #6773
- Password login request for SFTP: #2966
- `russh`: https://github.com/Eugeny/russh
- `russh-sftp`: https://github.com/AspectUnk/russh-sftp

Contributor guide

Open the contributing guide

Research direction

Start by reading the current implementation under core/services/sftp/ and the existing behavior tests, then trace how the openssh transport, fastpool connection pool, authentication, known-hosts strategies, and error mapping work. Done means replacing the transport with russh and russh-sftp while preserving the public API and capabilities, passing behavior tests on Linux and Windows, and updating the Unix-only documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, networking
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.