KMS root keys: state the no-rotation compromise, its blast radius, and the indirection that would make recovery possible

Start with .agent/ROOT-KEY-DESIGN.md and the cited KMS derivation, contract, measurement, and security-documentation files. Review options (1) and (2), then record the maintainer decision on those options and the intended position on (3) and (4) in the appropriate docs; done means the compromise model and recovery limits are explicit.

Class: DESIGN. This is not a bug report and nothing here is a vulnerability. It is a request to make an existing design position explicit, to price it, and to decide whether the cheapest of four mitigations is worth taking.

What the design currently is

Every app secret dstack issues is a pure function of one of two root keys and an app identity tuple. There is no version, epoch, generation or key-id input anywhere in the tree — a repo-wide grep over dstack/kms, the contracts and docs/ finds nothing but ACME credential rotation and VMM log rotation.

root-ca.key  (P-256, random at bootstrap, dstack/kms/src/onboard_service.rs:458)
├─ derive_p256_key_pair(root_ca.key, [app_id, "app-ca"])                        main_service.rs:335
├─ derive_dh_secret(root_ca.key, [app_id, instance_id, "app-disk-crypt-key"])   main_service.rs:380
├─ derive_dh_secret(root_ca.key, [app_id, "env-encrypt-key"])                   main_service.rs:384
└─ public_key_der() => app_compose.key_provider_id => mr_config_id => MRCONFIGID

root-k256.key  (secp256k1, random at bootstrap, onboard_service.rs:460)
└─ derive_key(root_k256, [app_id, "app-key"], 32)                               crypto.rs:15-16
     └─ IKM for every guest-agent GetKey at every path/domain

Installation of a new root is refused unconditionally: onboard_service.rs:115 and :144 both bail! if root-ca.key or root-k256.key exists. Replication (Onboard.Onboard → GetKmsKey, main_service.rs:492-501) copies the two private keys verbatim over RA-TLS. DstackKms.kmsInfo records {k256Pubkey, caPubkey, quote, eventlog} (DstackKms.sol:19-26) but isKmsAllowed never reads it (DstackKms.sol:255-278) — the on-chain record of which root is authoritative is documentation for relying parties, not an enforced gate.

Nearest existing record: #552 ("Static HKDF salt RATLS with no key versioning"), which docs/security/public-security-reports.md:31 parks with "Key versioning and rotation require a broader compatibility design." This issue is that broader design discussion.

The steelman — what the choice buys

Worth stating first, because these are real and the design should not be reversed casually:

1. No key distribution and no key database. The KMS holds two private keys and nothing else. There is no per-app key table to back up, corrupt, lose, replicate inconsistently or leak in a different shape. A new replica needs the root and nothing more.
2. Reproducible identity. app_id is an ERC1967Proxy address (DstackKms.sol:192) and the key is a pure function of it. An app that reboots on a different host three years later gets the same key with no lookup and no trust in the host to hand it the right one.
3. Attestable derivation. "Which key does app X get" is answerable by anyone holding the root pubkey — which is exactly what the dstack-kms-issued chain and kmsInfo exist for. A stateful KMS would have to be trusted to answer honestly.
4. No rollback surface. No counter or epoch that a hostile host can roll back by restoring an old disk image — the failure class already recorded twice in this codebase (GHSA-w9q3, GHSA-3643). Statelessness removes it entirely.

What it costs, concretely

1. Rotation moves every leaf, because identity is not an input that distinguishes roots

Run against the production kdf functions (harness validated first against the two golden vectors already in the tree, ra-tls/src/kdf.rs:175 and kms/src/crypto.rs:148):

== rotate the root: every leaf moves, identity inputs unchanged ==
app_ca   d28c45013377c638 -> 1a494de425d80d29  changed=true
env_key  0ec3dffce49a5beb -> 7500d314a567482b  changed=true
disk_key 5697330c3eb2ef95 -> 6923f0e73f6be6d7  changed=true
k256     14452120e868aef8 -> 48423d6453273784  changed=true
app_id fed to all four was byte-identical: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

2. Rotation also changes every app's measurement, which is the part that is easy to miss

app_compose.key_provider_id is, for a KMS-backed app, the root CA public key DER (pinned at dstack-util/src/system_setup.rs:2390-2403), and it is folded into the launch measurement at dstack-types/src/mr_config.rs:46-64:

hasher.update(compose_hash);
hasher.update(app_id);
hasher.update([key_provider_kind_byte(*key_provider)]);
hasher.update(key_provider_id);        // <- the root CA pubkey
// -> mr_config_id -> TDX MRCONFIGID -> mr_aggregated

Demonstrated with the production MrConfig::to_mr_config_id:

compose_hash and app_id are byte-identical in both rows.
mr_config_id under the OLD root CA: 027c18632d45f79b44d3a144a30994ee33a0e7ca8f6dc11e9076d11a5651918d4c00...
mr_config_id under the NEW root CA: 02ab09f8663c22e87dae588ba8d91a665e0226550d515809fc8cd6616969fd6d8200...
changed = true

So a rotation does not merely re-key applications; it changes the quote they present and therefore the identity the app contract sees. Every pinned app must be re-deployed with a new compose hash, not just re-keyed.

3. The artifact list

Must be re-encrypted (data loss if skipped):

• every LUKS2 data disk (/dev/vdb, PARTLABEL dstack-data) — the passphrase is hex(disk_crypt_key) (system_setup.rs:3039) and there is no second keyslot;
• every .encrypted-env blob, re-encrypted client-side under the new GetAppEnvEncryptPubKey;
• anything an app itself encrypted under a GetKey-derived key at any path — dstack has no inventory of these and cannot enumerate them.

Must be re-registered on chain:

• DstackKms.setKmsInfo(k256Pubkey, caPubkey, quote, eventlog) — one tx;
• every app's new compose hash into its DstackApp.addComposeHash, per §2 above;
• kmsAllowedAggregatedMrs / kmsAllowedDeviceIds for the new KMS CVM.

Must be re-pinned off chain:

• every guest's .host-shared/.appkeys.json (ca_cert, KeyProvider::Kms.pubkey) — written once at boot (system_setup.rs:3027), never refreshed;
• every gateway's gateway-ca.cert, the sole mTLS anchor for the WaveKV cluster (gateway/src/kv/https_client.rs:219-238) — rewritten only at gateway restart, so a rotation splits the cluster until every node restarts;
• vmm-cli.py trusted_signers (vmm/src/vmm-cli.py:1686-1713), dstack-util encrypt --kms-pubkey, get-keys --root-ca;
• every out-of-band relying party holding the old k256Pubkey. No shipped SDK verifies the dstack-kms-issued chain (Rust/Python/Go/JS all removed their verifiers), so dstack does not know who these are.

Does not change: app_id, registeredApps, allowedDeviceIds, allowedOsImages, gatewayAppId, appImplementation, WireGuard keys, ACME material.

4. The recovery story today

If the root is believed compromised, the complete set of actions available is:

1. removeKmsAggregatedMr(mr) / removeKmsDevice(deviceId) — gates future boots and onboards only. A KMS already holding the root keeps serving it; docs/specification.md:406 records the non-retroactivity for apps and it holds identically here.
2. Stop serving the instance — denies key release to every app, because there is no second root to fail over to.
3. Bootstrap a new KMS and pay §3 in full.

An operator cannot: revoke the old root for already-deployed guests (they pin ca_cert at boot and never re-check; no CRL, no OCSP, and kmsInfo is consulted by no gate); invalidate signatures the old root already produced (the legacy dstack-env-encrypt-pubkey envelope has no timestamp and no expiry); or recover any ciphertext.

There is also no forward secrecy, by construction: env_crypt_key is a static X25519 secret keyed on app_id alone, and the host retains .encrypted-env. An attacker who images /dev/vdb today and obtains the root in five years decrypts it.

Improvement directions

Four options, separated honestly by cost. (1) and (2) are cheap and useful on their own; (3) and (4) are migration events.

(1) Write the compromise model down. Cheap, no code.
A docs/security/ section stating: the root is never rotated; compromise is unrecoverable; operators should keep an off-CVM record of which root served which app. Today the docs describe replication and never say what happens if it fails. This is the option I would take first regardless of the others.

(2) Make a retired root detectable, even though it cannot be un-issued. Small.
Have the guest compare the KMS's k256_pubkey against DstackKms.kmsInfo — or a new isKmsRootRetired(bytes32) view — at GetAppKey time. This recovers no data, but it converts "silently talking to a revoked root" into a boot failure, which is the difference between a compromise and a catastrophe. It also gives kmsInfo an enforcement role it does not currently have.
Trade-off: adds a chain read to the boot path, on which #1259 already counts eight serialized round trips; and it hard-fails boots if kmsInfo was never populated — docs/onchain-governance.md never mentions setKmsInfo, so some deployments have it empty.

(3) Add a root epoch to the derivation info, and let the KMS hold two roots. Migration event.
LP(epoch) ‖ LP(app_id) ‖ … makes "app A under root 1" and "app A under root 2" distinguishable KDF inputs. A dual-root KMS can then answer GetAppKey with both the old and the new disk key in one response, and the guest can cryptsetup luksAddKey the new one and re-encrypt env in place — turning re-deployment into migration. The shape already exists in-tree: ra-tls/src/api_v1.rs:33-44,96-121 adopted exactly this (distinct salt, context tag, length-prefixed fields) for the guest API.
Trade-off: every deployed key is epoch 0 and must stay byte-identical, so this is additive-only and needs the same golden-vector discipline kms/src/crypto.rs already uses. It also re-introduces a small piece of KMS state — which costs property (4) of the steelman, though an epoch that only ever increases and is published on chain is not rollback-sensitive in the way a counter is.
And it does not solve §2 on its own.

(4) Stop using the CA public key as key_provider_id. The structural fix for §2.
Use a stable KMS identifier instead — the DstackKms contract address is the natural candidate, since it already determines every app_id. This decouples an app's measurement from the root's key, which is the single change that turns rotation from "re-deploy every app" into "re-key every app".
Trade-off: changes mr_config_id for every existing app exactly once, i.e. it costs one flag-day to buy the ability never to need another. It also weakens the pin: today key_provider_id binds the app to that specific root; a contract address binds it to whichever root that contract currently blesses, which only helps if (2) is in place to make kmsInfo meaningful.

What I am not asking for

Not for the derivation to become stateful, and not for rotation to be implemented now. The useful outcome of this issue is a maintainer decision on (1) and (2), and a recorded position on whether (3)+(4) are ever intended — so that the cost above is paid knowingly.

Full analysis, derivation tree, replication gates and reproduction harness: .agent/ROOT-KEY-DESIGN.md.