Dstack-TEE / Dstack-TEE/private-ai-gateway
Attestation doesn't bind login material: the serving CVM runs a dev image (is_dev: true) and provisions root authorized_keys from unmeasured user_config
- Dominant language
- Rust
- Stars
- 32
- Forks
- 8
- Avg merge
- 11h 9m
- Merged PRs (30d)
- 36
Description
A relying party can verify the full attestation chain for
`api.redpill.ai` — TDX quote, `report_data` binding, `compose_hash`, `app_id`,
a complete RTMR3 replay — and still have no way to determine who can log into
the CVM that handles their plaintext prompts. (With `supported_e2ee_versions:
[]` this gateway is not doing ACI E2EE decryption at all — it terminates TLS and
the request body is plaintext inside the trust boundary.)
To be precise about the two facts, since they do different work: the input that
decides *which* credentials get installed is `user_config`, which is never
measured. `is_dev` decides something narrower — whether the image contains a
login stack at all — and it is computed and surfaced but never gated.
I encountered this while building a third-party ACI client. The deployment I
observed is running commit `b7ca97ae` (the commit hash is in the server's own
attestation, under `attestation.source_provenance.repo_commit`).
### What I ran
```bash
curl -s https://api.redpill.ai/v1/aci/attestation \
| jq '.attestation.evidence.app_compose | fromjson
| {allowed_envs, public_logs, public_sysinfo,
prelaunch_reads_user_config:
(.pre_launch_script | test("ssh_authorized_keys"))}'
```
```json
{
"allowed_envs": [
"PRIVATE_AI_GATEWAY_ADMIN_TOKEN", "PRIVATE_AI_GATEWAY_CONTROL_URL",
"PRIVATE_AI_GATEWAY_CONTROL_TOKEN", "DSTACK_ROOT_PUBLIC_KEY",
"CLOUDFLARE_API_TOKEN", "CERTBOT_EMAIL",
"NODE_EXPORTER_USER", "NODE_EXPORTER_PASSWORD_HASH"
],
"public_logs": true,
"public_sysinfo": true,
"prelaunch_reads_user_config": true
}
```
The measured `pre_launch_script` (13,166 bytes, covered by `compose_hash`) does
this at its line 280 — matching
[`dstack-examples/phala-cloud-prelaunch-script/prelaunch.sh#L310-L312`](https://github.com/Dstack-TEE/dstack-examples/blob/d3fb09c2b1984d1537c0fad1112c39888b610a27/phala-cloud-prelaunch-script/prelaunch.sh#L310-L312):
```bash
if [[ -f /dstack/user_config ]] && jq empty /dstack/user_config 2>/dev/null; then
if [[ $(jq 'has("ssh_authorized_keys")' /dstack/user_config) == "true" ]]; then
jq -j '.ssh_authorized_keys' /dstack/user_config >> /home/root/.ssh/authorized_keys
```
### Why the attestation can't answer the question
**`/dstack/user_config` is host-written and never measured.** Your own docs, at
[`docs/security/cvm-boundaries.md#L133`](https://github.com/Dstack-TEE/dstack/blob/87e4285271109ade84552d71e1ed654d1884ffba/docs/security/cvm-boundaries.md#L133):
> This is an optional application-specific configuration file that applications
> inside the CVM can access. dstack OS simply stores it at
> `/dstack/.host-shared/.user-config` **without any measurement or additional
> processing**, unless `requirements.launch_token_hash` is set in
> app-compose.json — in that case the guest reads the launch token from JSON
> path `dstack.launch_token` in this file and fails closed at boot, before key
> provisioning, unless its SHA-256 matches the pinned hash.
The host writes it via `UpgradeApp`
([`vmm/src/main_service.rs#L658-L662`](https://github.com/Dstack-TEE/dstack/blob/87e4285271109ade84552d71e1ed654d1884ffba/dstack/vmm/src/main_service.rs#L658-L662))
and the guest copies it in without hashing
([`system_setup.rs#L309`](https://github.com/Dstack-TEE/dstack/blob/87e4285271109ade84552d71e1ed654d1884ffba/dstack/dstack-util/src/system_setup.rs#L309)).
In the RTMR3 event log this deployment publishes, no event carries `user_config`
or any digest of it — the full sequence is `system-preparing`, `app-id`,
`compose-hash`, `instance-id`, `boot-mr-done`, `mr-kms`, `os-image-hash`,
`key-provider`, `storage-fs`, `system-ready`. I am reporting what the log shows
rather than asserting a negative about every call site in the tree.
**`launch_token_hash` does not close this**, and I want to be accurate rather
than alarmist about it: it binds exactly one JSON pointer,
`/dstack/launch_token`
([`system_setup.rs#L940`](https://github.com/Dstack-TEE/dstack/blob/87e4285271109ade84552d71e1ed654d1884ffba/dstack/dstack-util/src/system_setup.rs#L940)).
Even with it set and satisfied, `ssh_authorized_keys` elsewhere in the same file
is unconstrained.
The surface is broader than SSH: the same script reads `.default_gateway_domain`
from the same unmeasured file (`prelaunch.sh#L336-L337`). SSH is simply the
sharpest illustration.
Worth separating clearly, because it decides who should act on this: **this is
dstack platform behaviour, and I am citing it rather than reporting it.** The
`user_config` design is documented and deliberate, and dstack already documents
the adjacent property that privileged in-CVM code can append measured events
after boot
([`docs/security/security-best-practices.md#L192-L194`](https://github.com/Dstack-TEE/dstack/blob/87e4285271109ade84552d71e1ed654d1884ffba/docs/security/security-best-practices.md#L192-L194)).
I also want to be exact about provenance: the SSH-provisioning script is **not**
in the dstack repo — it is Phala Cloud's prelaunch script from
`Dstack-TEE/dstack-examples`, which is what makes this a question about *this
deployment* rather than about the platform.
One note on how this differs from reports closed under the existing threat
model: the deciding input crosses the boundary **from the host**, before any
in-guest compromise. This is not "assume an attacker already has code running
inside the CVM" — it is the untrusted side of the documented boundary reaching a
measured, root-privileged script without appearing in any measurement.
**And the deployed image is a dev image — provable from the artifact, not from
anyone's word for it.** `os_image_hash` is content-addressed
(`os_image_hash = sha256(sha256sum.txt)`), so the published image self-verifies
as the one the hardware measured:
```bash
H=de9c74f0c85d0820ce075cb4a99f8e39f7b681be632907c5bf8bdc95ea72feb9 # attested
curl -sO https://download.dstack.org/os-images/mr_$H.tar.gz
tar -xzOf mr_$H.tar.gz sha256sum.txt | sha256sum
# de9c74f0c85d0820ce075cb4a99f8e39f7b681be632907c5bf8bdc95ea72feb9 ← matches
tar -xzOf mr_$H.tar.gz metadata.json
```
```json
{
"version": "0.5.9",
"git_revision": "e3655d1390feee3736476f4bda35c4354b4a12fc",
"cmdline": "console=ttyS0 init=/init panic=1 ... dstack.rootfs_hash=b171b92b...",
"is_dev": true
}
```
`is_dev: true`, and `git_revision` pins meta-dstack **v0.5.9**, whose
`dstack-rootfs-dev.inc` at that exact commit is four lines:
```
include dstack-rootfs-base.inc
IMAGE_INSTALL += "packagegroup-core-ssh-openssh strace tcpdump gdb gdbserver vim"
EXTRA_IMAGE_FEATURES += "debug-tweaks tools-profile"
```
The production recipe alongside it is `IMAGE_FEATURES += "nologin"`, which is
what removes the login stack. This image was built from the dev recipe, and its
kernel cmdline carries `console=ttyS0`.
**Limit of what I checked, stated plainly:** the tarball contains
`ovmf.fd`, `bzImage`, `initramfs.cpio.gz` and `metadata.json` — not the rootfs,
so I did not read `/usr/sbin/sshd` out of the image bytes. The rootfs is bound
transitively (`os_image_hash` → `metadata.json` → cmdline `dstack.rootfs_hash` →
dm-verity root hash), so its contents are pinned by the attestation; I simply
could not download them. The claim rests on the build recipe at the pinned
`git_revision`, not on an inspection of the shipped filesystem.
This also matches what this repo already says. From
`docs/providers/phala-direct/review.md#L38-L41`, unchanged since 2026-06-11:
> `production_os_image` is now decided from dstack's published image metadata
> (bound to the attested `os_image_hash`), so dev images are flagged today
> (**the deployed fleet is all `is_dev: true`**). Remaining: decide whether to
> **gate** on it (reject non-prod) for a strict release tier, vs. the current
> record-only behavior.
A production recipe exists in the same released line — `dstack-rootfs-prod.inc`
at that tag is `include dstack-rootfs-base.inc` plus
`IMAGE_FEATURES += "nologin"`, which is what strips the login stack. So this is
not a request for new work: the 0.5.x line already ships both recipes, and this
deployment was built from the dev one.
The clearest statement of *why* that matters is your own, from the newer mkosi
prod profile on master. I cite it for the reasoning only — I am aware that tree
is not in any released tag and did not build this image:
[`os/mkosi/mkosi.profiles/prod/mkosi.conf#L23-L25`](https://github.com/Dstack-TEE/dstack/blob/87e4285271109ade84552d71e1ed654d1884ffba/os/mkosi/mkosi.profiles/prod/mkosi.conf#L23-L25)
is exactly the property I want and states it better than I could:
> In the production image the login stack must be structurally absent from the
> verity-measured rootfs, not merely disabled by mutable state: the serial port
> is a chardev owned by the untrusted host.
To be clear about what I am and am not asking: I am not asking for a change to
the dev image, which is doing its job. I am asking about the choice to serve
production inference from one — and that choice belongs to this repo, which is
why I am filing here rather than on `dstack`.
### Pre-conceding your side
This is documented, intentional behaviour on both counts, and your threat model
already says development settings are auditable rather than production-safe, with
verifiers expected to reject them. That is precisely the gap: `os_image_is_dev`
is computed and surfaced but never gated
([`verifier/src/verification.rs#L909`](https://github.com/Dstack-TEE/dstack/blob/87e4285271109ade84552d71e1ed654d1884ffba/dstack/verifier/src/verification.rs#L909)
assigns it into the report). In that file `is_dev` appears exactly twice — once
copied in, once assigned out — and neither use rejects; I checked the verifier,
not every consumer in the workspace. And no claim covers interactive-access
provisioning at all. A relying party told to "reject dev settings" isn't handed
what it needs to do so.
I'd also rather not overstate the SSH half. On the meta-dstack/Yocto path that
built this image, password authentication is disabled by the
`disable-password-auth.conf` drop-in, and OpenSSH's first-value-wins `Include`
ordering means that drop-in beats what `debug-tweaks` edits. So this is
public-key access for whoever holds the operator key, not an open door, and I
know it was already addressed in `meta-dstack#44`. **This is not a re-report of
that issue.** My point is about what the attested document conveys to a third
party, which is unchanged by how the login is authenticated.
### The practical effect
I set out to offer users a route where no human other than the requester can see
their prompt. Everything I can verify says the host can't read enclave memory.
To be precise about what I can and cannot tell from the attestation, because the
distinction matters: I *can* see that this compose provisions interactive
credentials — the measured `pre_launch_script` says so plainly, and that is the
system working. What no measurement tells me is **which keys are installed,
whether the resulting `authorized_keys` is non-empty, and whether an SSH daemon
is listening** — because the deciding input arrives through an unmeasured file.
I want to be equally clear that I have **not** demonstrated a reachable root
shell: I did not observe a listening daemon or a reachable port, and the script
itself neither installs nor starts one. My claim is narrower and, I think,
harder to dismiss: *the attestation does not bind the login material*, so a
relying party cannot evaluate this either way.
### Not a duplicate
Not `#92` (`X-Upstream-Verification` routing) or its remediations `#99`/`#102`/
`#103` — this is about interactive access to the serving CVM, not upstream route
selection, and needs a different fix. Not items 3, 8, 13 or 19 of
`docs/reviews/aci-spec-conformance-gaps.md` — those cover E2EE defaults, §6
response authentication, the Chutes body hash, and stale live-E2E scripts. I
searched open and closed issues for `user_config` measurement and
`ssh_authorized_keys` and found nothing.
### What I'm asking for
Ordered by how much each buys a relying party per unit of work:
1. **A data point for the open question in your `review.md`.** You wrote
"Remaining: decide whether to gate on it (reject non-prod) for a strict
release tier." I am not claiming you overlooked this — I am answering it from
the other side: here is a relying party who cannot make the judgement without
the gate, because record-only means every client must independently know
which measurements mean "dev". A strict tier that rejects `is_dev` would
settle it.
2. **Serve production inference from a production image.** `dstack-rootfs-prod.inc`
ships in the same 0.5.x line as the dev recipe this image was built from, so
this is a build-target change rather than new work.
3. **A claim covering interactive-access provisioning** — whether the running
configuration provisions any login credential — so this is answerable from
the attestation instead of by reading a 13 KB shell script.
4. **A way to bind more of `user_config` than `launch_token_hash` reaches.**
I want to frame this as a feature request, not a bug: the docs are explicit
that `user_config` is unmeasured by design, and `launch_token_hash` is the
opt-in binding mechanism. The gap is that it binds exactly one pointer, so
there is no way to bind `ssh_authorized_keys` or `default_gateway_domain`.
A key allowlist, or a digest over a declared subset, would close it without
the cost of measuring the whole file — which I recognise is real, since it
would make routine per-instance edits attestation-affecting and RTMR3 feeds
the full-replay authorization paths.
5. **Digest-pin `dstacktee/dstack-verifier:latest`.** It is the only mutable tag
among the four images in the compose. `compose_hash` covers the tag *string*,
not the digest it resolves to, so this one container can change under a
compose hash that stays constant.
6. **Consider `public_logs: false` and `public_sysinfo: false`**, and dropping
`DSTACK_ROOT_PUBLIC_KEY` from `allowed_envs` if the deployment does not need
operator SSH. Listed last because these are deployment preferences rather
than gaps in what is verifiable.
There is a related gap I want to flag without over-claiming, because it affects
the whole TCB rather than just interactive access: `source_provenance` gives
`repo_commit` but leaves **`image_digest` and `image_provenance` null**. Combined
with (5), a relying party can see which source commit is claimed but cannot
verify that the running containers were built from it. I do not know whether
those fields are unimplemented or simply unset here, which is why this is a
question rather than an ask.
I'm happy to send a PR for (1) or (3) if the shape is agreed.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with docs/providers/phala-direct/review.md and dstack/verifier/src/verification.rs to trace how production-image status reaches the attestation report. Then review the cited user_config and prelaunch behavior to define the required boundary. Done should let a relying party determine whether the serving image and login material meet the intended production policy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, shell
- Domain
- authentication, cloud, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100