Dstack-TEE / Dstack-TEE/dstack

vmm-cli: encrypted env vars fail in three ways that all surface as "variable is required"

Open
#832 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
544
Forks
96
Avg merge
17h 57m
Merged PRs (30d)
117

Description

Hit all three while deploying a CVM whose app needs one encrypted environment
variable — a DNS provider token. Each failed in a way that pointed somewhere
other than the actual cause, and together they cost about an hour and three
redeploys. Filing as one issue since they are all the same feature.

## 1. `--env-file` accepts a file it cannot parse, silently

`parse_env_file` expects `KEY=Value` lines and skips everything else. Given a
JSON file it parses zero variables and returns an empty dict, and `deploy`
carries on as if no `--env-file` had been passed:

```json
[{"key": "CLOUDFLARE_API_TOKEN", "value": "..."}]
```

Deploy prints nothing unusual. The VM boots, the app starts, and the failure
finally surfaces inside the container as

```
Error: CLOUDFLARE_API_TOKEN environment variable is required
```

which reads like an application bug. `[{"key": …, "value": …}]` is a reasonable
guess at the format, and it happens to be the shape `deploy` builds internally
before encrypting.

Rejecting a file that yields no variables would be enough — an `--env-file` that
sets nothing is never intentional.

## 2. `compose` cannot set `allowed_envs`

An encrypted variable is only injected if `app_compose.allowed_envs` names it,
but `vmm-cli compose` has no flag for that list, so it always writes
`"allowed_envs": []`. The only way to set it is to edit the generated JSON by
hand.

That is worse than it sounds, because the app id is derived from that file. So
adding a variable name changes the app id, which invalidates anything already
keyed to it — for a `dstack-gateway` app that means the
`_dstack-app-address.` TXT record has to be updated before traffic works
again.

The symptom is identical to #1 (`… environment variable is required`), so from
the container's point of view "wrong file format" and "variable not allowed" are
indistinguishable.

Something like `compose --allowed-env NAME` (repeatable), or defaulting the list
to the keys of `--env-file` when one is given, would remove the hand edit and the
app-id churn that follows it.

## 3. `deploy --kms-url` sets two different things at once

It is used both for the host-side call that fetches the app's env encryption key
and for `params["kms_urls"]`, which is what the *guest* is configured to talk to.
Those are rarely the same address.

In a local lab the KMS listens on `127.0.0.1:13501` on the host, while the guest
reaches it via QEMU user networking as `kms.1022.dstack.org` → `10.0.2.2`. The
vmm config already has both:

```toml
kms_url = "https://localhost:13501" # host side
kms_urls = ["https://kms.1022.dstack.org:13501"] # guest side
```

Passing `--kms-url https://localhost:13501` — the only value that works for the
encryption step — overwrites the guest's list too, and the CVM then reboot-loops:

```
dstack-prepare.sh[334]: Error: Failed to request app keys
0: Failed to get app keys from KMS
1: Failed to get temp ca cert
...
6: Connection refused (os error 111)
[FAILED] Failed to start dstack-prepare.service
```

No message connects that to the flag; the VM simply cycles between `booting` and
`requesting app keys` until you read the console log.

The workaround is to deploy without `--env-file`, then

```bash
vmm-cli.py update-env --env-file env --kms-url https://localhost:13501
```

and restart, because `update_vm_env` uses the URL only to fetch the key and does
not write it into the VM config. It works, but it means the documented one-step
path is the broken one.

A separate flag for the encryption endpoint — or falling back to the vmm's own
`kms_url` when only the guest list should change — would fix it. Erroring when
the given URL is a loopback address would at least fail at the right moment.

## Environment

`vmm-cli.py` from `main`, local lab (vmm + kms + gateway on one TDX host),
guest image `dstack-0.6.0`.

Contributor guide

Open the contributing guide

Research direction

Start in vmm-cli.py, reading parse_env_file, deploy, update_vm_env, and the compose path that writes generated JSON. Trace how allowed_envs and the host and guest KMS URLs are populated. Done means empty or invalid env files fail clearly, compose can set allowed_envs, and one-step encrypted-env deployment preserves separate KMS endpoints.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, cli, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.