Dstack-TEE / Dstack-TEE/private-ai-gateway

Attestation doesn't bind login material: the serving CVM runs a dev image (is_dev: true) and provisions root authorized_keys from unmeasured user_config

Open
#119 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32
Forks
8
Avg merge
11h 9m
Merged PRs (30d)
36

Description

A relying party can verify the full attestation chain for
`api.redpill.ai` — TDX quote, `report_data` binding, `compose_hash`, `app_id`,
a complete RTMR3 replay — and still have no way to determine who can log into
the CVM that handles their plaintext prompts. (With `supported_e2ee_versions:
[]` this gateway is not doing ACI E2EE decryption at all — it terminates TLS and
the request body is plaintext inside the trust boundary.)

To be precise about the two facts, since they do different work: the input that
decides *which* credentials get installed is `user_config`, which is never
measured. `is_dev` decides something narrower — whether the image contains a
login stack at all — and it is computed and surfaced but never gated.

I encountered this while building a third-party ACI client. The deployment I
observed is running commit `b7ca97ae` (the commit hash is in the server's own
attestation, under `attestation.source_provenance.repo_commit`).

### What I ran

```bash
curl -s https://api.redpill.ai/v1/aci/attestation \
| jq '.attestation.evidence.app_compose | fromjson
| {allowed_envs, public_logs, public_sysinfo,
prelaunch_reads_user_config:
(.pre_launch_script | test("ssh_authorized_keys"))}'
```

```json
{
"allowed_envs": [
"PRIVATE_AI_GATEWAY_ADMIN_TOKEN", "PRIVATE_AI_GATEWAY_CONTROL_URL",
"PRIVATE_AI_GATEWAY_CONTROL_TOKEN", "DSTACK_ROOT_PUBLIC_KEY",
"CLOUDFLARE_API_TOKEN", "CERTBOT_EMAIL",
"NODE_EXPORTER_USER", "NODE_EXPORTER_PASSWORD_HASH"
],
"public_logs": true,
"public_sysinfo": true,
"prelaunch_reads_user_config": true
}
```

The measured `pre_launch_script` (13,166 bytes, covered by `compose_hash`) does
this at its line 280 — matching
[`dstack-examples/phala-cloud-prelaunch-script/prelaunch.sh#L310-L312`](https://github.com/Dstack-TEE/dstack-examples/blob/d3fb09c2b1984d1537c0fad1112c39888b610a27/phala-cloud-prelaunch-script/prelaunch.sh#L310-L312):

```bash
if [[ -f /dstack/user_config ]] && jq empty /dstack/user_config 2>/dev/null; then
if [[ $(jq 'has("ssh_authorized_keys")' /dstack/user_config) == "true" ]]; then
jq -j '.ssh_authorized_keys' /dstack/user_config >> /home/root/.ssh/authorized_keys
```

### Why the attestation can't answer the question

**`/dstack/user_config` is host-written and never measured.** Your own docs, at
[`docs/security/cvm-boundaries.md#L133`](https://github.com/Dstack-TEE/dstack/blob/87e4285271109ade84552d71e1ed654d1884ffba/docs/security/cvm-boundaries.md#L133):

> This is an optional application-specific configuration file that applications
> inside the CVM can access. dstack OS simply stores it at
> `/dstack/.host-shared/.user-config` **without any measurement or additional
> processing**, unless `requirements.launch_token_hash` is set in
> app-compose.json — in that case the guest reads the launch token from JSON
> path `dstack.launch_token` in this file and fails closed at boot, before key
> provisioning, unless its SHA-256 matches the pinned hash.

The host writes it via `UpgradeApp`
([`vmm/src/main_service.rs#L658-L662`](https://github.com/Dstack-TEE/dstack/blob/87e4285271109ade84552d71e1ed654d1884ffba/dstack/vmm/src/main_service.rs#L658-L662))
and the guest copies it in without hashing
([`system_setup.rs#L309`](https://github.com/Dstack-TEE/dstack/blob/87e4285271109ade84552d71e1ed654d1884ffba/dstack/dstack-util/src/system_setup.rs#L309)).
In the RTMR3 event log this deployment publishes, no event carries `user_config`
or any digest of it — the full sequence is `system-preparing`, `app-id`,
`compose-hash`, `instance-id`, `boot-mr-done`, `mr-kms`, `os-image-hash`,
`key-provider`, `storage-fs`, `system-ready`. I am reporting what the log shows
rather than asserting a negative about every call site in the tree.

**`launch_token_hash` does not close this**, and I want to be accurate rather
than alarmist about it: it binds exactly one JSON pointer,
`/dstack/launch_token`
([`system_setup.rs#L940`](https://github.com/Dstack-TEE/dstack/blob/87e4285271109ade84552d71e1ed654d1884ffba/dstack/dstack-util/src/system_setup.rs#L940)).
Even with it set and satisfied, `ssh_authorized_keys` elsewhere in the same file
is unconstrained.

The surface is broader than SSH: the same script reads `.default_gateway_domain`
from the same unmeasured file (`prelaunch.sh#L336-L337`). SSH is simply the
sharpest illustration.

Worth separating clearly, because it decides who should act on this: **this is
dstack platform behaviour, and I am citing it rather than reporting it.** The
`user_config` design is documented and deliberate, and dstack already documents
the adjacent property that privileged in-CVM code can append measured events
after boot
([`docs/security/security-best-practices.md#L192-L194`](https://github.com/Dstack-TEE/dstack/blob/87e4285271109ade84552d71e1ed654d1884ffba/docs/security/security-best-practices.md#L192-L194)).
I also want to be exact about provenance: the SSH-provisioning script is **not**
in the dstack repo — it is Phala Cloud's prelaunch script from
`Dstack-TEE/dstack-examples`, which is what makes this a question about *this
deployment* rather than about the platform.

One note on how this differs from reports closed under the existing threat
model: the deciding input crosses the boundary **from the host**, before any
in-guest compromise. This is not "assume an attacker already has code running
inside the CVM" — it is the untrusted side of the documented boundary reaching a
measured, root-privileged script without appearing in any measurement.

**And the deployed image is a dev image — provable from the artifact, not from
anyone's word for it.** `os_image_hash` is content-addressed
(`os_image_hash = sha256(sha256sum.txt)`), so the published image self-verifies
as the one the hardware measured:

```bash
H=de9c74f0c85d0820ce075cb4a99f8e39f7b681be632907c5bf8bdc95ea72feb9 # attested
curl -sO https://download.dstack.org/os-images/mr_$H.tar.gz
tar -xzOf mr_$H.tar.gz sha256sum.txt | sha256sum
# de9c74f0c85d0820ce075cb4a99f8e39f7b681be632907c5bf8bdc95ea72feb9 ← matches
tar -xzOf mr_$H.tar.gz metadata.json
```

```json
{
"version": "0.5.9",
"git_revision": "e3655d1390feee3736476f4bda35c4354b4a12fc",
"cmdline": "console=ttyS0 init=/init panic=1 ... dstack.rootfs_hash=b171b92b...",
"is_dev": true
}
```

`is_dev: true`, and `git_revision` pins meta-dstack **v0.5.9**, whose
`dstack-rootfs-dev.inc` at that exact commit is four lines:

```
include dstack-rootfs-base.inc

IMAGE_INSTALL += "packagegroup-core-ssh-openssh strace tcpdump gdb gdbserver vim"
EXTRA_IMAGE_FEATURES += "debug-tweaks tools-profile"
```

The production recipe alongside it is `IMAGE_FEATURES += "nologin"`, which is
what removes the login stack. This image was built from the dev recipe, and its
kernel cmdline carries `console=ttyS0`.

**Limit of what I checked, stated plainly:** the tarball contains
`ovmf.fd`, `bzImage`, `initramfs.cpio.gz` and `metadata.json` — not the rootfs,
so I did not read `/usr/sbin/sshd` out of the image bytes. The rootfs is bound
transitively (`os_image_hash` → `metadata.json` → cmdline `dstack.rootfs_hash` →
dm-verity root hash), so its contents are pinned by the attestation; I simply
could not download them. The claim rests on the build recipe at the pinned
`git_revision`, not on an inspection of the shipped filesystem.

This also matches what this repo already says. From
`docs/providers/phala-direct/review.md#L38-L41`, unchanged since 2026-06-11:

> `production_os_image` is now decided from dstack's published image metadata
> (bound to the attested `os_image_hash`), so dev images are flagged today
> (**the deployed fleet is all `is_dev: true`**). Remaining: decide whether to
> **gate** on it (reject non-prod) for a strict release tier, vs. the current
> record-only behavior.

A production recipe exists in the same released line — `dstack-rootfs-prod.inc`
at that tag is `include dstack-rootfs-base.inc` plus
`IMAGE_FEATURES += "nologin"`, which is what strips the login stack. So this is
not a request for new work: the 0.5.x line already ships both recipes, and this
deployment was built from the dev one.

The clearest statement of *why* that matters is your own, from the newer mkosi
prod profile on master. I cite it for the reasoning only — I am aware that tree
is not in any released tag and did not build this image:
[`os/mkosi/mkosi.profiles/prod/mkosi.conf#L23-L25`](https://github.com/Dstack-TEE/dstack/blob/87e4285271109ade84552d71e1ed654d1884ffba/os/mkosi/mkosi.profiles/prod/mkosi.conf#L23-L25)
is exactly the property I want and states it better than I could:

> In the production image the login stack must be structurally absent from the
> verity-measured rootfs, not merely disabled by mutable state: the serial port
> is a chardev owned by the untrusted host.

To be clear about what I am and am not asking: I am not asking for a change to
the dev image, which is doing its job. I am asking about the choice to serve
production inference from one — and that choice belongs to this repo, which is
why I am filing here rather than on `dstack`.

### Pre-conceding your side

This is documented, intentional behaviour on both counts, and your threat model
already says development settings are auditable rather than production-safe, with
verifiers expected to reject them. That is precisely the gap: `os_image_is_dev`
is computed and surfaced but never gated
([`verifier/src/verification.rs#L909`](https://github.com/Dstack-TEE/dstack/blob/87e4285271109ade84552d71e1ed654d1884ffba/dstack/verifier/src/verification.rs#L909)
assigns it into the report). In that file `is_dev` appears exactly twice — once
copied in, once assigned out — and neither use rejects; I checked the verifier,
not every consumer in the workspace. And no claim covers interactive-access
provisioning at all. A relying party told to "reject dev settings" isn't handed
what it needs to do so.

I'd also rather not overstate the SSH half. On the meta-dstack/Yocto path that
built this image, password authentication is disabled by the
`disable-password-auth.conf` drop-in, and OpenSSH's first-value-wins `Include`
ordering means that drop-in beats what `debug-tweaks` edits. So this is
public-key access for whoever holds the operator key, not an open door, and I
know it was already addressed in `meta-dstack#44`. **This is not a re-report of
that issue.** My point is about what the attested document conveys to a third
party, which is unchanged by how the login is authenticated.

### The practical effect

I set out to offer users a route where no human other than the requester can see
their prompt. Everything I can verify says the host can't read enclave memory.

To be precise about what I can and cannot tell from the attestation, because the
distinction matters: I *can* see that this compose provisions interactive
credentials — the measured `pre_launch_script` says so plainly, and that is the
system working. What no measurement tells me is **which keys are installed,
whether the resulting `authorized_keys` is non-empty, and whether an SSH daemon
is listening** — because the deciding input arrives through an unmeasured file.
I want to be equally clear that I have **not** demonstrated a reachable root
shell: I did not observe a listening daemon or a reachable port, and the script
itself neither installs nor starts one. My claim is narrower and, I think,
harder to dismiss: *the attestation does not bind the login material*, so a
relying party cannot evaluate this either way.

### Not a duplicate

Not `#92` (`X-Upstream-Verification` routing) or its remediations `#99`/`#102`/
`#103` — this is about interactive access to the serving CVM, not upstream route
selection, and needs a different fix. Not items 3, 8, 13 or 19 of
`docs/reviews/aci-spec-conformance-gaps.md` — those cover E2EE defaults, §6
response authentication, the Chutes body hash, and stale live-E2E scripts. I
searched open and closed issues for `user_config` measurement and
`ssh_authorized_keys` and found nothing.

### What I'm asking for

Ordered by how much each buys a relying party per unit of work:

1. **A data point for the open question in your `review.md`.** You wrote
"Remaining: decide whether to gate on it (reject non-prod) for a strict
release tier." I am not claiming you overlooked this — I am answering it from
the other side: here is a relying party who cannot make the judgement without
the gate, because record-only means every client must independently know
which measurements mean "dev". A strict tier that rejects `is_dev` would
settle it.
2. **Serve production inference from a production image.** `dstack-rootfs-prod.inc`
ships in the same 0.5.x line as the dev recipe this image was built from, so
this is a build-target change rather than new work.
3. **A claim covering interactive-access provisioning** — whether the running
configuration provisions any login credential — so this is answerable from
the attestation instead of by reading a 13 KB shell script.
4. **A way to bind more of `user_config` than `launch_token_hash` reaches.**
I want to frame this as a feature request, not a bug: the docs are explicit
that `user_config` is unmeasured by design, and `launch_token_hash` is the
opt-in binding mechanism. The gap is that it binds exactly one pointer, so
there is no way to bind `ssh_authorized_keys` or `default_gateway_domain`.
A key allowlist, or a digest over a declared subset, would close it without
the cost of measuring the whole file — which I recognise is real, since it
would make routine per-instance edits attestation-affecting and RTMR3 feeds
the full-replay authorization paths.
5. **Digest-pin `dstacktee/dstack-verifier:latest`.** It is the only mutable tag
among the four images in the compose. `compose_hash` covers the tag *string*,
not the digest it resolves to, so this one container can change under a
compose hash that stays constant.
6. **Consider `public_logs: false` and `public_sysinfo: false`**, and dropping
`DSTACK_ROOT_PUBLIC_KEY` from `allowed_envs` if the deployment does not need
operator SSH. Listed last because these are deployment preferences rather
than gaps in what is verifiable.

There is a related gap I want to flag without over-claiming, because it affects
the whole TCB rather than just interactive access: `source_provenance` gives
`repo_commit` but leaves **`image_digest` and `image_provenance` null**. Combined
with (5), a relying party can see which source commit is claimed but cannot
verify that the running containers were built from it. I do not know whether
those fields are unimplemented or simply unset here, which is why this is a
question rather than an ask.

I'm happy to send a PR for (1) or (3) if the shape is agreed.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with docs/providers/phala-direct/review.md and dstack/verifier/src/verification.rs to trace how production-image status reaches the attestation report. Then review the cited user_config and prelaunch behavior to define the required boundary. Done should let a relying party determine whether the serving image and login material meet the intended production policy.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, shell
Domain
authentication, cloud, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.