NVIDIA / NVIDIA/OpenShell

feat: FIPS 140-3 compliance path for TLS and SSH crypto stack

Aperta
#900 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

state:stale
Lingua principale
Rust
Stelle
8.7k
Fork
1.3k
Merge medio
2g 7h
PR unite (30g)
243

Descrizione

Problem Statement

OpenShell's TLS and SSH subsystems use non-FIPS-validated cryptographic libraries and default to non-FIPS-approved algorithms. On FIPS-enabled clusters (common in government, defense, and regulated-industry Kubernetes deployments), this makes OpenShell non-compliant from an audit perspective -- even though the kernel does not block the operations at runtime.

The sandbox has been split into two crates: openshell-supervisor-process (SSH, process lifecycle) and openshell-supervisor-network (proxy, L7, TLS MITM). References below reflect this split.

Specifically:

  • TLS (all connections): Uses ring 0.17 via rustls. ring has no FIPS validation path. Default cipher suites include ChaCha20-Poly1305 and X25519 key exchange, neither of which are FIPS-approved.
  • SSH (sandbox transport): Uses russh 0.62 with a mix of aws-lc-rs, ed25519-dalek, and curve25519-dalek. The sandbox SSH server hardcodes Ed25519 host keys (openshell-supervisor-process/src/ssh.rs:52). Default negotiation prefers ChaCha20-Poly1305 and Curve25519 key exchange.
  • PKI (certificate generation): Uses rcgen 0.13 backed by ring. The default algorithm (ECDSA P-256) is FIPS-approved, but the implementation module is not validated.
  • Credential encryption (at rest): The openshell-driver-db-credstore crate performs envelope encryption using ring directly (AES-256-GCM via ring::aead, key material via ring::rand::SystemRandom). This is the most audit-exposed crypto surface outside of TLS.
  • JWT signing: Sandbox JWTs use EdDSA (Ed25519) for signing and validation. Ed25519 is approved only under FIPS 186-5 with a validated module.

FIPS-enabled RHEL 9 / OpenShift 4.x clusters enforce FIPS 140-3 via system-wide crypto policies. Processes using non-validated crypto modules fail compliance audits regardless of the algorithms selected. There are no existing FIPS-related issues in the tracker.

This is complementary to #899 (Platform mode / restricted SCC support) -- FIPS clusters are a subset of the managed Kubernetes deployments that issue addresses.

Proposed Design

Add a workspace-level fips Cargo feature flag that switches the crypto backend from ring to aws-lc-rs in FIPS mode (CMVP certificate #4631), restricts algorithm negotiation to FIPS-approved algorithms only, and documents the SSH layer's validation gap.

Phase 1: Feature-flagged FIPS for TLS + PKI + credential encryption + EdDSA JWT decision

Crypto provider switch -- Five binary-facing sites install the rustls CryptoProvider and would switch based on the feature flag:

// Current (all five sites):
rustls::crypto::ring::default_provider().install_default()

// With --features fips:
rustls::crypto::aws_lc_rs::default_provider().install_default()

The five sites are:

  • openshell-server/src/cli.rs:222
  • openshell-cli/src/main.rs:2154
  • openshell-sandbox/src/main.rs:522 (debug-rpc)
  • openshell-sandbox/src/main.rs:568 (main supervisor)
  • openshell-sdk/src/transport.rs:186 (SDK crate)

Additionally, openshell-server/src/tls.rs:21 has a direct ring import (use rustls::crypto::ring::sign;) that needs editing even after the provider swap.

Workspace dependency changes:

# Current:
rustls = { version = "0.23", default-features = false, features = ["std", "logging", "tls12", "ring"] }
tokio-rustls = { version = "0.26", default-features = false, features = ["logging", "tls12", "ring"] }
rcgen = { version = "0.13", features = ["crypto"] }

# With fips feature:
rustls = { version = "0.23", default-features = false, features = ["std", "logging", "tls12", "aws_lc_rs", "fips"] }
tokio-rustls = { version = "0.26", default-features = false, features = ["logging", "tls12", "aws-lc-rs"] }
rcgen = { version = "0.13", default-features = false, features = ["aws_lc_rs", "pem"] }

TLS cipher suite restriction -- In FIPS mode, configure the provider to exclude ChaCha20-Poly1305 cipher suites and X25519 key exchange, allowing only:

  • TLS 1.3: TLS13_AES_256_GCM_SHA384, TLS13_AES_128_GCM_SHA256
  • TLS 1.2: TLS_ECDHE_ECDSA_WITH_AES_*_GCM_SHA*, TLS_ECDHE_RSA_WITH_AES_*_GCM_SHA*
  • Key exchange: ECDH-P256, ECDH-P384 (no X25519)

SSH algorithm restriction -- Change the sandbox SSH server's host key from Ed25519 to ECDSA-P256 and configure russh::server::Config::preferred / russh::client::Config::preferred to exclude non-FIPS algorithms:

  • Host keys: ecdsa-sha2-nistp256, ecdsa-sha2-nistp384, rsa-sha2-256, rsa-sha2-512 (no ssh-ed25519)
  • Key exchange: ecdh-sha2-nistp256, ecdh-sha2-nistp384, diffie-hellman-group14-sha256, diffie-hellman-group16-sha512 (no Curve25519, no post-quantum mlkem)
  • Ciphers: aes256-gcm@openssh.com, aes128-gcm@openssh.com, aes256-ctr, aes128-ctr (no ChaCha20-Poly1305)

Credential encryption backend switch -- The openshell-driver-db-credstore crate performs envelope encryption (AES-256-GCM KEK/DEK, key material via ring's SystemRandom). This is the most audit-exposed crypto surface and must be included in Phase 1. The ring imports (ring::aead, ring::rand) switch to aws-lc-rs equivalents under the fips feature flag.

EdDSA JWT decision -- Sandbox JWTs use EdDSA (sandbox_jwt.rs:117, validation at :177, :181-182), keyed by Ed25519 (bootstrap/jwt.rs:42 PKCS_ED25519). Ed25519 is approved only under FIPS 186-5 with a validated module. Decision needed: pin to ES256 in FIPS mode, or confirm coverage in the target aws-lc-fips certificate. Wire-format change either way.

Transitive dependency updates -- Several workspace dependencies pull in rustls and/or ring. Each needs feature flags updated for the FIPS build:

  • reqwest: switch from rustls-tls-native-roots to using the globally-installed CryptoProvider
  • sqlx with tls-rustls-ring-native-roots: feature name explicitly pins ring, needs updating
  • kube with rustls-tls: needs feature flag update
  • hyper-rustls: switch from ring to aws-lc-rs feature
  • tokio-tungstenite: switch from rustls-tls-native-roots to aws-lc-rs backend

rustls 0.21 leg -- rustls 0.21.12 exists in the dependency graph via hyper-rustls 0.24.2 (AWS SDK path). The 0.21 leg predates pluggable CryptoProvider, so the workspace feature flag won't reach it. Either unify onto rustls 0.23 or account for a second unvalidated ring copy.

Phase 1 testing -- No test today would catch a crypto backend swap or a regression in algorithm restriction. Phase 1 needs negotiated-suite assertions, not just build-flag changes:

  • Integration tests that verify the TLS cipher suites actually negotiated in FIPS mode are restricted to the approved set
  • Tests that the ring::default_provider() sites are replaced by aws_lc_rs::default_provider() when the fips feature is active
  • Negative tests confirming that non-FIPS algorithms (ChaCha20-Poly1305, X25519, Ed25519 host keys) are rejected during negotiation in FIPS mode

Go SDK -- GOFIPS140=v1.0.0 build flag for go 1.24+/1.26 toolchain is the cheapest FIPS win for the Go SDK. The Python SDK's BoringSSL gap is documented but out of scope for this issue.

UBI glibc supervisor image -- The supervisor currently uses an Alpine/musl static build (Dockerfile.supervisor). A musl static build cannot dynamically link a system OpenSSL FIPS provider. A UBI-based FIPS image variant needs a separate glibc build. The feat(build): add glibc-static supervisor libc variant commit is a step toward this, but a full UBI glibc variant is needed for the system-OpenSSL FIPS path.

Phase 2 (deferred): SSH transport FIPS validation

Phase 1 restricts SSH to FIPS-approved algorithms but the underlying implementations (ed25519-dalek, p256, aes from RustCrypto) remain non-validated modules. This is a known gap. The SSH transport only operates within the cluster's mTLS boundary (gateway-to-sandbox), providing defense-in-depth rather than being the primary trust boundary.

The NSSH1 nonce/HMAC handshake has been removed. SSH authentication is now unconditional (auth_none and auth_publickey both return Auth::Accept), with trust established via the unix socket (SO_PEERCRED) or the abstract socket in sidecar topology. This strengthens the Phase 2 deferral argument: the SSH layer carries no authentication of its own and operates entirely inside the mTLS boundary, making it even more clearly defense-in-depth.

If strict auditors require validated modules for the SSH layer, Phase 2 options include:

  • Upstream russh support for aws-lc-rs as its crypto backend
  • Replacing the embedded russh server with an OpenSSH subprocess (significant architecture change given the deep integration at ssh.rs -- 1700+ lines of process spawning, PTY management, channel handling, SFTP subsystem)

Note: ssh_tunnel.rs was deleted (#1029). russh is now 0.62 (was 0.57 in the original investigation).

Scope boundaries:

  • The fips feature is off by default -- current behavior is preserved
  • Phase 1 achieves FIPS-validated crypto for all TLS operations (the external-facing attack surface), credential-at-rest encryption, and FIPS-approved algorithms for SSH
  • Phase 1 includes the EdDSA JWT decision (pin to ES256 or confirm aws-lc-rs coverage)
  • Phase 1 includes negotiated-suite test assertions
  • Phase 1 explicitly documents the SSH validation gap
  • Phase 2 is deferred to actual audit requirements
Alternatives Considered
  1. System OpenSSL for everything -- Replace rustls with the openssl crate and russh with libssh2 or OpenSSH subprocess. True FIPS validation for all operations via RHEL 9's OpenSSL 3.x (CMVP #4282). Rejected for Phase 1: massive rewrite, loses rustls memory safety guarantees, adds system library dependency, and significantly complicates cross-platform builds.

  2. Partial compliance with documented exceptions -- FIPS for TLS only, document SSH as internal-only transport. This is essentially what Phase 1 achieves, but framed as the complete solution rather than a stepping stone. May not satisfy strict auditors.

  3. No FIPS support -- Require FIPS-mode clusters to use custom crypto policy exceptions for OpenShell pods. Not viable for enterprise adoption in regulated environments.

  4. gVisor RuntimeClass -- gVisor provides its own syscall interception and could theoretically handle crypto at the runtime level. Not applicable -- gVisor intercepts syscalls, not userspace crypto library calls.

Agent Investigation

Investigation performed with a coding agent pointed at the repo. Skills loaded: create-spike, generate-sandbox-policy. The agent traced every crypto dependency, configuration point, and algorithm choice across the workspace.

Crypto dependency map

The TLS and SSH subsystems use different crypto backends -- a critical finding for the migration path:

TLS path (all connections):
  rustls 0.23.38 -> ring 0.17
  rcgen 0.13 -> ring 0.17
  rustls-webpki -> ring 0.17
  quinn-proto -> ring 0.17

SSH path (sandbox transport):
  russh 0.62 -> aws-lc-rs
  russh 0.62 -> ed25519-dalek -> curve25519-dalek
  russh 0.62 -> aes, cbc, ctr (RustCrypto symmetric)
  russh 0.62 -> p256, p384, p521
  russh 0.62 -> libcrux-ml-kem (post-quantum)

Credential encryption path:
  openshell-driver-db-credstore -> ring 0.17 (aead, rand)
  openshell-driver-db-credstore -> sha2 (RustCrypto, for key ID)

Code references

Location Description
Cargo.toml:38-39 Workspace rustls/tokio-rustls pinned to ring feature
Cargo.toml:41 rcgen 0.13 with crypto feature (ring backend)
Cargo.toml:90 reqwest with rustls-tls-native-roots
Cargo.toml:97 tokio-tungstenite with rustls-tls-native-roots
Cargo.toml:105 jsonwebtoken 10 with aws_lc_rs (non-FIPS build)
Cargo.toml:107 ring 0.17 as a direct workspace dependency
Cargo.toml:126 sqlx with tls-rustls-ring-native-roots -- feature name pins ring explicitly
Cargo.toml:129 kube with rustls-tls
openshell-server/src/cli.rs:222 ring::default_provider().install_default()
openshell-cli/src/main.rs:2154 ring::default_provider().install_default()
openshell-sandbox/src/main.rs:522, :568 ring::default_provider().install_default() -- two sites (debug-rpc and main supervisor)
openshell-sdk/src/transport.rs:186 ring::default_provider() -- SDK crate, fifth binary-facing site
openshell-server/src/tls.rs:21 use rustls::crypto::ring::sign; -- direct ring import, needs editing even after provider swap
openshell-server/src/tls.rs:280, :285 Gateway mTLS ServerConfig (no cipher suite or kx customization)
openshell-cli/src/tls.rs:220 CLI mTLS ClientConfig (no cipher suite customization)
openshell-cli/src/tls.rs:268, openshell-sdk/src/transport.rs:186 supported_verify_schemes() reads ring's signature_verification_algorithms directly
openshell-supervisor-network/src/l7/tls.rs:156 MITM proxy ServerConfig
openshell-supervisor-network/src/l7/tls.rs:210 MITM proxy upstream ClientConfig
openshell-supervisor-network/src/l7/tls.rs:44, :56 MITM ephemeral CA keygen + self_signed (rcgen/ring)
openshell-supervisor-network/src/l7/tls.rs:116, :123 MITM leaf keygen + signed_by (rcgen/ring)
openshell-supervisor-process/src/ssh.rs:52 PrivateKey::random(&mut rng, Algorithm::Ed25519) -- hardcoded Ed25519 host key
openshell-supervisor-process/src/ssh.rs:54 russh::server::Config { auth_rejection_time, ..Default::default() } -- preferred unset
openshell-server/src/grpc/sandbox.rs:1780 exec_ssh_client_config() -- russh client Config with Default, preferred unset
openshell-bootstrap/src/pki.rs:55, :75, :93 CA / server / client keygen via rcgen::KeyPair::generate() (ECDSA P-256 via ring)
openshell-driver-db-credstore/src/lib.rs:29-30 ring::aead and ring::rand imports
openshell-driver-db-credstore/src/lib.rs:40 const ALGORITHM: &str = "AES-256-GCM"
openshell-driver-db-credstore/src/lib.rs:594 KEK generation via ring SystemRandom
openshell-driver-db-credstore/src/lib.rs:663 Per-value DEK generation
openshell-driver-db-credstore/src/lib.rs:726, :742 seal_in_place_append_tag / open_in_place
openshell-driver-db-credstore/src/lib.rs:907 random_bytes_status() -- key and nonce material from ring's RNG
openshell-driver-db-credstore/src/lib.rs:915 key_id() -- SHA-256 via RustCrypto sha2
openshell-server/src/auth/sandbox_jwt.rs:117 EdDSA JWT signing
openshell-server/src/auth/sandbox_jwt.rs:177, :181-182 EdDSA validation pinned to Algorithm::EdDSA
openshell-bootstrap/src/jwt.rs:42 PKCS_ED25519 key generation
openshell-bootstrap/src/jwt.rs:61 SHA-256 key ID
deploy/docker/Dockerfile.gateway:22 Gateway base: gcr.io/distroless/cc-debian13:nonroot
deploy/docker/Dockerfile.supervisor:22 Supervisor base: alpine:3.22 (musl target)
deploy/docker/Dockerfile.ci:9 CI base: nvcr.io/nvidia/base/ubuntu:noble-20251013

Non-FIPS algorithm inventory

Operation Current Algorithm FIPS? FIPS Alternative Controlling Code
TLS 1.3 cipher ChaCha20-Poly1305 (in default list) No AES-256-GCM, AES-128-GCM CryptoProvider cipher suite list
TLS key exchange X25519 (in default list) No ECDH-P256, ECDH-P384 CryptoProvider kx_group list
TLS crypto module ring 0.17 No aws-lc-rs (CMVP #4631) Cargo.toml feature flags
SSH host key Ed25519 (hardcoded) No ECDSA-P256, ECDSA-P384 ssh.rs:52
SSH key exchange curve25519-sha256 (default preferred) No ecdh-sha2-nistp256/384 russh::Config::preferred
SSH cipher chacha20-poly1305 (default preferred) No aes256-gcm, aes128-gcm russh::Config::preferred
SSH KEX (PQ) mlkem768x25519 No Remove from preference list russh::Config::preferred
SSH crypto module ed25519-dalek, RustCrypto AES, p256 No aws-lc-fips-sys (requires upstream russh changes) russh internals
PKI key generation ECDSA P-256 via ring Algorithm OK, module not validated ECDSA P-256 via aws-lc-rs rcgen backend feature
Credential KEK/DEK AES-256-GCM via ring Algorithm OK, module not validated AES-256-GCM via aws-lc-rs openshell-driver-db-credstore
Credential RNG ring SystemRandom Module not validated aws-lc-rs DRBG openshell-driver-db-credstore
Credential key ID SHA-256 via RustCrypto sha2 Algorithm OK, module not validated SHA-256 via aws-lc-rs credstore lib.rs:915
Sandbox JWT signing Ed25519/EdDSA Approved under FIPS 186-5 only with validated module ES256 or confirm aws-lc-rs EdDSA coverage sandbox_jwt.rs, bootstrap/jwt.rs

Existing FIPS awareness: Zero. The only mention of "FIPS" in the codebase is in an OCSF schema JSON referencing NIST FIPS 199 (information classification standard, unrelated to crypto).

Dead dependency: openshell-server/Cargo.toml:102 declares hmac = "0.12" with no remaining .rs usage in the server crate -- leftover from the removed NSSH1 handshake. Should be cleaned up regardless of FIPS work.

SSH RNG note: rand::rng() (userspace ChaCha12, OS-seeded) generates the SSH host key at ssh.rs:51. FIPS posture wants a validated DRBG for key material.

SHA-1 note: openshell-supervisor-network/src/l7/rest.rs:2274 uses SHA-1 for RFC 6455 Sec-WebSocket-Accept. This is protocol-mandated, not a security function. Permissible but will trip scanners; needs a documented exemption.

Test-only RSA note: rsa 0.9 at openshell-server/Cargo.toml:131 is a dev-dependency for test RS256 keys. Not in the shipped dependency graph.

Feature flag patterns: The codebase already uses workspace-level feature propagation (bundled-z3, dev-settings) and platform-conditional compilation via #[cfg(target_os = "linux")]. A #[cfg(feature = "fips")] pattern would be consistent.

Risks & open questions:

  1. aws-lc-rs FIPS build requires CMake + Go, adding build toolchain complexity
  2. russh's internal crypto (ed25519-dalek, p256, RustCrypto AES) is not FIPS-validated regardless of algorithm selection -- Phase 1 documents this gap
  3. Does russh have upstream plans for an aws-lc-rs or FIPS backend?
  4. Cross-compilation from macOS to linux/amd64 for FIPS container builds may require remote builds
  5. SSH host key change from Ed25519 to ECDSA-P256 changes fingerprint -- sandboxes are ephemeral, so should not cause persistent trust issues
  6. The current tree has aws-lc-rs but no aws-lc-fips-sys -- the FIPS variant is a distinct sys crate. Verify that the target aws-lc-fips-sys build matches CMVP #4631
  7. Transitive deps (reqwest, sqlx, tokio-tungstenite, hyper-rustls, kube) each need verification with aws-lc-rs provider
  8. Single fips feature flag vs separate fips-tls/fips-ssh for phased rollout?
  9. The musl supervisor image blocks the system-OpenSSL FIPS path; a UBI glibc variant is needed for production FIPS compliance
  10. The rustls 0.21 leg (via hyper-rustls 0.24.2, AWS SDK path) predates pluggable CryptoProvider and will carry an unvalidated ring copy even after the workspace feature flag switch
  11. EdDSA JWT wire-format change: pinning to ES256 in FIPS mode changes the JWT signing algorithm, requiring coordinated rollout between gateway and sandbox components
Checklist
  • I've reviewed existing issues and the architecture docs
  • This is a design proposal, not a "please build this" request

Guida per i contributori

Apri la guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Direzione di ricerca

Inizia dalle voci delle dipendenze del workspace in Cargo.toml e dai cinque punti dei provider rustls elencati in openshell-server/src/cli.rs, openshell-cli/src/main.rs, openshell-sandbox/src/main.rs e openshell-sdk/src/transport.rs. Poi esamina openshell-server/src/tls.rs, openshell-supervisor-process/src/ssh.rs, sandbox_jwt.rs, bootstrap/jwt.rs e gli import di ring di credential-store. Per considerarlo completato sono necessari un backend feature-gated, la negoziazione di algoritmi approvati, test delle suite negoziate, una decisione su JWT e la documentazione della lacuna nella validazione SSH.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
rust
Ambito
backend, cryptography, security
Tipo di issue
Funzionalità
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Attiva
Chiarezza
Abbastanza chiara
Idoneità per principianti
28/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.