Devolutions / Devolutions/sspi-rs
CredSSP NLA pub_key_auth mismatch with gnome-remote-desktop server
- Dominant language
- Rust
- Stars
- 83
- Forks
- 50
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 9
Description
## Summary
When connecting to a [gnome-remote-desktop](https://gitlab.gnome.org/GNOME/gnome-remote-desktop) (GRD) server with NLA enabled (the default for user-session mode on Fedora 44), CredSSP fails on the server with `nla_decrypt_public_key_hash: Could not verify server's hash`. Microsoft's official Windows RDP client connects to the same server successfully. The TLS handshake succeeds — the failure is at the CredSSP layer.
GRD's RDP server is built on FreeRDP-server, so this is structurally an sspi-rs ↔ FreeRDP-server interop bug.
## Reported via
- [GlassHaven/Haven#109](https://github.com/GlassHaven/Haven/issues/109#issuecomment-4322381205) — user @agevlakh on Fedora 44, gnome-remote-desktop user-mode session.
- Haven uses ironrdp 0.8.0 (which depends on sspi 0.19.2).
## Server-side log from the failure
```
[ERROR][com.freerdp.core.nla] - [nla_decrypt_public_key_hash]: Could not verify server's hash
[ERROR][com.freerdp.core.transport] - [transport_accept_nla]: client authentication failure
[ERROR][com.freerdp.core.peer] - [peer_recv_callback_internal]: CONNECTION_STATE_NEGO - rdp_server_accept_nego() fail
```
i.e. the server received the client's pubKeyAuth, decrypted it, but the SHA-256 it computed independently from `magic + nonce + server_public_key` didn't match.
## What I checked
- sspi-rs `encrypt_public_key_hash` (sspi 0.19.2 `credssp/mod.rs:1240`) hashes `SHA256(CLIENT_SERVER_HASH_MAGIC + nonce + public_key)` where `public_key` is whatever the caller passes.
- ironrdp passes the SubjectPublicKey BIT STRING bits (extracted from the TLS server cert via `x509-cert::Certificate::subject_public_key_info.subject_public_key.as_bytes()`).
- FreeRDP server (`libfreerdp/core/nla.c::nla_encrypt_public_key_hash`) hashes the same shape, where `public_key` is the output of `freerdp_certificate_get_public_key()` — implemented as OpenSSL's `i2d_PublicKey(EVP_PKEY*)`.
- Empirically verified for an RSA-2048 self-signed cert: `i2d_PublicKey(rsa_key)` produces 270 bytes, the SubjectPublicKey BIT STRING content (post-`as_bytes()`) is also 270 bytes — **byte-identical** for RSA. Magic strings and nonce-size (32 bytes) match between sspi-rs and FreeRDP.
- ⇒ for RSA certs, SHA-256 inputs *should* be identical. But GRD says they aren't. So either there's a subtle off-by-one in the bytes (length-prefix included one side and not the other?) or the cert is non-RSA on agevlakh's setup or there's a CredSSP version-specific binding I'm missing.
## Couldn't reproduce locally
Tried Ubuntu Questing + GRD 49.0 (same major version as Fedora 44). My local GRD picks `SecurityProtocol(SSL)` rather than HYBRID/HYBRID_EX during X.224 nego (so NLA is never exercised), even though the source at `grd-session-rdp.c:1277-1279` explicitly sets `NlaSecurity=TRUE`/`RdpSecurity=FALSE`/`TlsSecurity=FALSE`. Even xfreerdp3 with `/sec:nla` against my GRD fails with `Protocol Security Negotiation Failure`. So my local GRD seems to have a runtime issue (TPM/SAM init?) that prevents it from honouring its own NLA-only config. Agevlakh's user-mode GRD on Fedora 44 evidently doesn't have that issue.
## Asks
1. Has Devolutions tested sspi-rs against gnome-remote-desktop / FreeRDP-server (3.x) recently? Any known interop trip wires?
2. Is there a CredSSP version > 6 binding shape (e.g. tls-server-end-point Channel Binding using TBSCertificate rather than raw subjectPublicKey) that sspi-rs negotiates which FreeRDP-server might encode differently?
3. If you can repro: a per-side hex-dump of the bytes fed into the SHA-256 update would settle this in one shot.
Happy to provide more data from the affected user once we know what to ask for. Will also be testing v5.24.48 of Haven (now with structured `sspi::ErrorKind` mapping) which should give us the inner sspi error variant the next time agevlakh tries — that may narrow it further.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.