Onboarding audit: cut 'time to first dstack app' from ~22 steps to a single script

## The problem

If you have a TDX host and want to run your first dstack app, the README says "deploy your own" and links to the deployment guide. Following that guide end-to-end takes around **22 ordered steps across two repos** before a `docker-compose.yaml` is reachable in a browser.

The first time I walked through it I had:

- Three terminals open (vmm, auth-simple, log tail)
- One browser tab on the KMS bootstrap page
- One text editor on `auth-config.json` that I edited four separate times
- A Cloudflare API token I had to provision because the gateway needs DNS-01 ACME
- A domain I had to own because the gateway URL pattern is `-.gateway.`

We can do better. This issue is about figuring out how.

## What's painful, concretely

A few moments where I sat there and thought "this should not be a step":

**The chicken-and-egg dance with the KMS allowlist.** On a fresh `auth-config.json`, `kms.mrAggregated` is empty, and `auth-simple` denies all KMS boots against an empty allowlist — but you can't know the value until the KMS CVM has booted. So the documented procedure is: deploy the KMS, watch bootstrap fail, call `Onboard.GetAttestationInfo` to read the hash, paste it into `auth-config.json`, retry. For a single-node KMS this gate is near-circular — it's the operator's KMS proving its own measurement to an allowlist the same operator just wrote — and removing it is a config default, not a hard problem (see the KMS section below).

**The "exit 1, edit the .env, run me again" pattern.** Three different scripts do this: `kms/dstack-app/deploy-simple.sh`, `gateway/dstack-app/deploy-to-vmm.sh`, `gateway/dstack-app/bootstrap-cluster.sh`. On the happy path that's three re-invocations just to get past prompts that could have been CLI flags or interactive prompts.

**Copy-paste a hash, then type `y`.** When you deploy the gateway, the script prints a compose hash, asks `Continue? [y/N]`, and the right answer is *not yet* — you're supposed to put the hash in `auth-config.json` first, then come back and type `y`. Same dance again when you deploy your first app. The script knows the hash. The auth file is on the same disk. It could just write it.

**Three terminals, no systemd.** vmm, auth-simple, and the log tail all run as foreground processes. There's no installed service, no `systemctl status`, no `journalctl`. Restarts after config edits are manual. Lifecycle is the operator's problem.

**Domain required everywhere.** The KMS bootstrap UI has a required "domain" text input. The gateway needs a real DNS-resolvable wildcard and a Cloudflare token. The app URL pattern bakes the gateway domain in. The only escape hatch today is `--port tcp:0.0.0.0:HOST:CTR`, which works but skips the gateway entirely, gives you plain HTTP, and isn't called out as a real path.

**Restart vmm to pick up new URLs.** `vmm.toml` has `kms_urls` and `gateway_urls`, both read at startup. So the order is: start vmm → deploy KMS → edit vmm.toml → restart vmm → deploy gateway → edit vmm.toml → restart vmm.

**Two repos.** `meta-dstack` owns the host-side build config and the OS image. `dstack` owns the services. For a hoster running a single TDX box, both repos are needed and the cross-references (`build.sh hostcfg`, `digest.txt`, `build-config.sh`) are easy to lose track of.

## What "good" could look like

Two tiers. **Tier 1 gets you to a running app with no domain and no gateway.** Tier 2 adds managed HTTPS + routing as a deliberate, separate step for people who want it.

### Tier 1 — first app (default)

A first-time user on a TDX+SGX host with Docker and nothing else:

```
$ sudo apt install dstack # package installs binaries + systemd units

$ sudo dstack init
✓ checked hardware: Intel TDX + SGX present
✓ downloaded guest image (dstack-0.5.7, digest matches)
✓ generated vmm.toml, kms.toml, auth-config.json
✓ configured single-node KMS auth (no measurement pre-registration needed)
✓ started dstack-vmm, dstack-auth-simple, gramine-sealing-key-provider
✓ deployed KMS CVM in Local-Key-Provider mode, auto-bootstrapped
✓ wired kms_urls, restarted dstack-vmm
→ VMM dashboard: https://:9080 (self-signed — click through the warning)

$ dstack run my-docker-compose.yaml
✓ computed compose hash → registered with auth-simple
✓ deployed app, waiting for boot
→ App URL: http://:9300/ (direct port mapping — no gateway, no domain)
```

The user owns no domain, holds no Cloudflare token, never opens a browser to bootstrap KMS, never edits a JSON file, never restarts a process by hand, and never sets up a gateway. The full security model still applies — real TDX attestation, KMS in a CVM, real `auth-simple` — we've just stopped asking the user to be the integration glue, and access is via direct host:port mapping instead of gateway routing.

### Tier 2 — managed HTTPS + routing (optional, separate)

When you want pretty `https://-.gateway.` URLs, automatic Let's Encrypt certs, and load-balanced routing, you set up the gateway as a second step:

```
$ sudo dstack gateway init # opt-in; this is where a domain + ACME provider come in
```

Keeping the gateway out of the default flow means the headline quickstart never blocks on DNS or a Cloudflare token, and the gateway's genuine complexity (wildcard DNS, ACME, WireGuard) is opt-in rather than mandatory. It also removes the contradiction of "deploy without a domain" — Tier 1 simply has no gateway, so there's no wildcard-hostname-to-resolve problem; it's direct host:port.

## Design decisions

These pin down the rest of the work. The first three are decided; flagging them so reviewers can object.

### 1. Two-tier onboarding: KMS-first, gateway optional

KMS is **not** optional — without it you don't get the real experience (persistent, upgradeable per-app keys derived in a TEE). So Tier 1 is "vmm + single-node KMS + your app via direct port", and the gateway is a separate Tier 2 step. This is the single decision that unblocks "deploy without a domain": the domain/DNS/ACME complexity all lives in the gateway, so making the gateway opt-in makes the domain opt-in for free.

### 2. Hardware: SGX required, fail fast

`dstack init` **refuses to run on a host without SGX** and exits with a clear message. KMS-in-CVM attestation depends on the Gramine SGX sealing key provider; silently degrading to host-mode KMS (no real attestation) would undermine the "full experience" promise. Evaluating on cloud TDX without SGX is explicitly not supported by `dstack init` — that's a conscious trade for not shipping a footgun.

### 3. TLS: self-signed by default; real domain only in the gateway tier

Self-signed certs are the only honest "no-domain" answer for a server-deployed product. mkcert-style "install the CA in your trust store" is great UX *when the issuer and the browser are on the same machine* — they're not, when dstack runs on a server and you browse from a laptop. We can't paper over that with a script.

- **Tier 1 surfaces** (VMM dashboard, KMS bootstrap, direct-port app access): self-signed cert with the host's IP, hostname, and `localhost` in the SAN. Browser warning; user clicks through. KMS already does exactly this today (`kms/src/onboard_service.rs`). Optional: download the dstack CA and install it on your laptop to silence the warning — documented per-OS, not scripted (the server/laptop gap makes it un-scriptable).
- **Tier 2 (gateway):** real domain + Cloudflare/Route53 token + Let's Encrypt DNS-01 (today's production path, kept as-is).
- **sslip.io** is the no-domain wildcard option for Tier 2 routing — `-..sslip.io` resolves cleanly without owning DNS. But it isn't on the Public Suffix List (verified against [publicsuffix.org's PSL](https://publicsuffix.org/list/public_suffix_list.dat)), so every `*.sslip.io` Let's Encrypt cert shares one global rate limit. So it's a documented "works, with a caveat" option, not the recommended default.

### 4. Process & packaging: systemd-native, installed as an OS package

The software ships as an OS package (apt/deb, dnf/rpm). The **package** owns installing and removing the binaries and the systemd units — `apt remove dstack` is the uninstall, not a bespoke subcommand:

- `dstack-vmm.service` — main VMM, `Restart=always`, logs via `journalctl -u dstack-vmm`.
- `dstack-auth-simple.service` — auth-simple webhook. Hot-reload of `auth-config.json` already exists.
- `gramine-sealing-key-provider.service` — SGX key provider. (Gateway, in Tier 2, brings its own unit.)

`dstack init` is then *deployment* bootstrap, not software install: generate configs, start services, bring up KMS, wire URLs. Its inverse is a deployment teardown (`dstack destroy` / `reset` — name TBD) that removes CVMs, generated configs, and keys but leaves the software installed. This separation is cleaner than an `init`/`uninstall` pair, which would conflate "install software" with "stand up a deployment".

Second-order wins: the `vmm.toml.kms_urls` restart pain becomes `systemctl restart dstack-vmm` driven by `dstack init` (no human in the loop); ordering (`After=`, `Requires=`) replaces the "edit → restart → deploy → edit → restart" dance; "three terminals" → one `journalctl -fu dstack-vmm`. We implicitly punt non-systemd distros (musl/Alpine, FreeBSD), which is fine — TDX hosts are almost universally Ubuntu/Debian/Fedora.

## KMS modes — what counts as "the full experience"

Three independent axes get conflated in the docs today, so it's worth naming them.

**Boot mode** (per `kms/README.md`):
- **Non-KMS Mode** — ephemeral per-boot keys, no persistence, no upgrades. `app-id == compose-hash`. Useful as a "show me a CVM run" demo target with zero infra. Not the quickstart bar.
- **Local-Key-Provider Mode** — SGX-sealed keys via Gramine, persistent, but `app-id == compose-hash` so upgrades are awkward. This is how the KMS itself runs.
- **KMS Mode** — full deterministic per-app keys derived from a KMS root, persistent + upgradeable, `app-id == app contract address`. This is what apps should run in.

**Auth backend:**
- **auth-mock** — always allow, demo/testing only.
- **auth-simple** — JSON allowlist, single-operator. Good fit for self-hosters.
- **auth-eth** — on-chain via smart contracts. The decentralized-governance path.

**Where the KMS runs:**
- **On the host** — current "dev deployment". No Gramine, no SGX, no real attestation of the KMS itself. Marked as "no security guarantees" in the docs.
- **In a CVM with Gramine** — current "production deployment". Real TDX attestation of the KMS. Requires SGX BIOS + the Gramine sealing key provider.

**Quickstart target:** KMS Mode (for apps) + KMS-in-CVM with Gramine (for the KMS itself) + auth-simple (for governance). That gives the full security story minus the blockchain — real attestation, real per-app key derivation, real upgrade path, single-operator authorization via JSON.

`auth-eth` is documented as the upgrade path: same `dstack init` flow, swap the auth backend. Non-KMS mode and Local-Key-Provider-for-apps stay as advanced examples in the docs, not as quickstart options.

**One config default makes the single-node path clean: `enforce_self_authorization = false`.** This is the fix for the chicken-and-egg above, and it's worth understanding precisely. With the default (`true`), the KMS self-attests to its own auth API before it will bootstrap: it builds its own boot info (`local_kms_boot_info`, which includes `mr_aggregated`) and POSTs it to `bootAuth/kms`, which `auth-simple` rejects until `kms.mrAggregated` is populated (`kms/src/main_service/upgrade_authority.rs:218`, `kms/auth-simple/index.ts:120`). For a single operator who owns the auth config, this self-gate is near-circular and buys nothing — turning it off lets the KMS bootstrap immediately with no measurement pre-registration. Crucially, this changes **only** the KMS proving *itself*:

- App authorization is untouched — apps still go through `bootAuth/app`, which checks the compose hash, not `mrAggregated`.
- App attestation is untouched — `GetAppKey`/`SignCert` still verify each requesting app's own TDX quote.
- The `mrAggregated` allowlist only does real work when a second KMS node replicates from the first (`ensure_kms_allowed`), which a single-node quickstart never does.

(For the record: `mr_aggregated` is `SHA256(mr_td ‖ rtmr0 ‖ rtmr1 ‖ rtmr2 ‖ rtmr3)` — `dstack-attest/src/attestation.rs:766` — so it can't simply be precomputed from `dstack-mr`, which only emits MRTD + RTMR0–2. RTMR3 folds in runtime measurements including the Gramine key-provider's MRENCLAVE. That's another reason to drop the gate for single-node rather than try to predict the value.)

The honest cost of picking "KMS-in-CVM + Gramine" as the default is that Gramine setup is itself painful today (see `docs/tutorials/gramine-key-provider.md`). If `dstack init` doesn't automate it, we've moved the friction rather than fixed it. So Gramine bring-up — pulling and starting `gramine-sealing-key-provider` as a systemd unit, pointing KMS at it, verifying — has to be part of `dstack init`.

## How to get there

Roughly three layers, in priority order. Layers 1–2 are Tier 1 (the headline goal); layer 3 is the optional gateway tier.

**1. The `dstack` CLI + `dstack init`.** A new Rust crate produces a single user-facing `dstack` binary (`init`, `run`, `ls`, `logs`, `destroy`, …) that talks to the VMM over the existing prpc interface (reusing `http-client` and the `*-rpc` proto crates — not a rewrite) and supersedes `vmm-cli.py`. **Rust, not Python** — the repo is already almost entirely Rust, a static binary drops straight into the OS package next to `dstack-vmm`, and it removes the Python-on-the-host friction (venv, system-Python pinning) that a self-hoster hits today. `dstack init` is the bring-up: render `vmm.toml`/`kms.toml`/`auth-config.json` from a few inputs (host IP, image version, mode flags); refuse on non-SGX hosts; start the systemd units; generate the single-node `kms.toml` with `enforce_self_authorization = false` so bootstrap doesn't gate on a measurement the operator hasn't seen yet; auto-derive the KMS bootstrap domain (host IP or `localhost`) so the browser step disappears (`kms/src/onboard_service.rs:367`). Idempotent re-runs.

**2. `dstack run` — eliminate the manual hash dance.** `dstack run ` wraps compose + register + deploy into one step: compute the compose hash, write it into `auth-config.json` for you, deploy, and expose the app via direct port mapping. The "exit 1, edit .env, re-run" pattern in the deploy scripts goes away — they fold into `dstack` subcommands.

**3. Tier-2 gateway as an opt-in step.** `dstack gateway init` brings up the gateway CVM for people who want managed HTTPS + routing. Gateway grows a self-signed no-domain mode (mirroring KMS) for local use; the real-domain → Let's Encrypt DNS-01 path stays for production; sslip.io is the documented no-domain wildcard option (with the shared-rate-limit caveat). All the domain/DNS/ACME complexity is confined here.

## Open questions

- **meta-dstack consolidation.** For a self-host quickstart, the host-side artifacts (`vmm.toml` template, systemd units, OS image tarball) could ship inside the `dstack` OS package rather than requiring a second repo clone + build. Worth doing in this issue's scope, or separate?
- **Teardown semantics.** What does `dstack destroy`/`reset` remove by default — CVMs + configs but keep KMS keys (so you can re-init against the same identity), or wipe everything? Probably a safe default + a `--purge` flag.
- **`vmm-cli.py` transition.** Hard-replace it with the `dstack` binary, or wrap it during a deprecation window? It has real users and scripts depending on it today.

## Action items

Tracking the work as a checklist here rather than as separate issues. Ordered by dependency — the critical path to Tier 1 (`apt install dstack` → `dstack init` → `dstack run`) is items 1–6; the rest is delivery polish and the opt-in gateway tier.

**Critical path to Tier 1**

- [x] **1. Spike + default: hands-off single-node KMS bootstrap.** ✅ **Validated on real TDX+SGX hardware** (see findings comment below). A single-node KMS CVM bootstraps fully hands-off (~30s) with `enforce_self_authorization = false` + a set `auto_bootstrap_domain` — no browser, no `mrAggregated` pre-registration, no manual step — and `bootAuth/app` + `GetAppKey`/`SignCert` checks are untouched. **Refinement:** the flag is *necessary but not sufficient* — bootstrap still needs a genesis TDX quote from the guest agent, so it only completes **inside a CVM** (a bare host fails at "Failed to get quote" regardless of the flag). Item #5's `init` therefore must run KMS in a CVM.
- [x] **2. systemd units + minimal package layout.** `dstack-vmm.service`, `dstack-auth-simple.service`, `gramine-sealing-key-provider.service` with `Restart=always`, journald logging, and `After=`/`Requires=` ordering. No CLI dependency; kills "three terminals" on its own.
- [ ] **3. Gramine sealing-key-provider bring-up automation.** Pull → start as the unit from #2 → point KMS at it → verify (`key-provider-build/run.sh`, `docs/tutorials/gramine-key-provider.md`). On the Tier 1 critical path because KMS-in-CVM + Gramine is the default.
- [x] **4. CLI crates + prpc client (two binaries).** `dstack` = the **client** (local or remote: connects to a local `vmm.sock` or a remote VMM over TLS + `--token`) — `run`/`ls`/`logs`/`info`/`upgrade`, plus `dstack init` = **scaffold a new app project** (`app-compose.yaml` + `.env`). `dstackup` = **host setup/lifecycle** (local + privileged). Reuse `ra-rpc` client + the `vmm-rpc` proto crate — not a rewrite. Ship read-only `ls`/`logs` first to prove the plumbing. (Precedent: kubeadm/kubectl, rustup/cargo.)
- [x] **5. `dstackup install` (host bring-up).** Render `vmm.toml`/`kms.toml`/`auth-allowlist.json` (host IP, image version, mode flags); SGX gate (refuse + clear message on non-SGX); **auto-pick free host ports and attach to an existing VMM rather than spawn a second**; start the units from #2; Gramine bring-up (#3); deploy the KMS CVM from the **pinned published image** with `enforce_self_authorization=false` + `auto_bootstrap_domain=`; poll `KMS.GetMeta` for readiness; wire `kms_urls` **per-deploy via the RPC (no VMM restart)**. Dashboard binds **`localhost` by default** (secure context over plain HTTP, no cert); `--expose ` mints a self-signed cert (SAN = the IP). Idempotent re-runs.
- [x] **6. `dstack run `.** Compute the compose hash → add it to the auth allowlist → **encrypt env vars natively in Rust (no browser `crypto.subtle`)** → deploy → expose via direct host:port. Retires the "exit 1, edit `.env`, re-run" + "copy-paste a hash, type `y`" dances; the deploy scripts fold into subcommands.

**Delivery + Tier 2**

- [x] **7. `dstackup destroy` / `reset`.** Host teardown (removes CVMs + generated configs; keep KMS keys by default + `--purge`).
- [ ] **8. OS package (deb first, rpm next).** Bundles binaries + units; `apt remove dstack` is the uninstall. The delivery vehicle that makes the Tier 1 headline real.
- [ ] **9. Tier 2: `dstack gateway init`.** Opt-in gateway CVM; gateway grows a self-signed no-domain mode mirroring KMS; real-domain → Let's Encrypt DNS-01 stays for production; sslip.io documented with the shared-rate-limit caveat.
- [ ] **10. Docs: two-tier quickstart rewrite.** Replace the ~22-step path; demote Non-KMS / Local-Key-Provider-for-apps to advanced examples; document `auth-eth` as the same flow with a swapped backend.

**Decisions — resolved**

- [x] **vmm-cli.py transition** (#4): **wrap-then-deprecate** — keep it working during a deprecation window while the `dstack` CLI supersedes it. It has live users.
- [x] **Teardown semantics** (#7): `dstack destroy`/`reset` **keeps KMS keys by default** (so you can re-init against the same identity); `--purge` wipes everything.
- [x] **meta-dstack consolidation** (#8): **keep the two repos separate for now** — may merge the host artifacts into the package later.
- [x] **Image source — KMS/gateway/verifier containers** (#5/#6): use the **prebuilt, pinned, reproducibly-built images already published to Docker Hub** — `dstacktee/dstack-kms`, `-gateway`, `-verifier` (versioned tags 0.5.4–0.5.11; built with `SOURCE_DATE_EPOCH` + pinned `DSTACK_REV`, with a build-provenance attestation pushed to the registry, so the digest is independently rebuildable from source). `dstack run`/`init` **pulls a pinned digest — not** the dev `dockerfile_inline` build-from-source compose. Pin to a current version and keep it fresh (the in-repo `kms/dstack-app/deploy-to-vmm.sh` digest is stale at 0.5.5 vs latest 0.5.11). The guest OS image stays separate (meta-dstack, per #8).
- [x] **Tier-1 auth backend** (#5): **reimplement the simple JSON-allowlist webhook in Rust** inside the `dstack` binary, instead of requiring the `bun`-based `auth-simple`. Keeps webhook→compose-hash allowlist + `enforce_self_authorization=false` (app governance survives, KMS self-bootstrap stays hands-off) while removing a runtime dependency — consistent with the Rust-native, no-extra-runtime goal.
- [x] **CLI shape** (#4): **two binaries** — `dstackup` (host setup/lifecycle, local + privileged) and `dstack` (client, local or remote), with `dstack init` reserved for **app-project scaffolding**. Mirrors kubeadm/kubectl. Setup commands are local-only; client commands work local or remote (TLS + token).
- [x] **Dashboard access** (#5): **localhost by default** — the VMM dashboard binds `127.0.0.1`; reached via SSH tunnel / on-box at `http://localhost:9080`, which is a secure context, so the browser deploy UI's `crypto.subtle` works with **no TLS and no cert** (and the Rust `dstack run` path never uses `crypto.subtle` at all). `dstackup install --expose ` is the opt-in that binds the IP and mints a self-signed cert (SAN = the IP). So self-signed-TLS-on-the-VMM-UI is an **opt-in feature, not a default blocker**.

Item #1 is validated (see findings comment). #5 (`dstackup install`) has a detailed implementation plan ready.