Dstack-TEE / Dstack-TEE/service-mesh
Remove the -parallelism=1 requirement (pre-allocate KMS nonces)
- Dominant language
- Go
- Stars
- 1
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
All deploys are stuck on `terraform apply -parallelism=1`, one cluster at a time. Anything concurrent — `-parallelism > 1` within one apply, or two applies at once — fails during app creation with:
```
[ERR-02-009] Failed to obtain app ID from centralized KMS:
duplicate key value violates unique constraint "ix_dstack_app_nonces_address"
```
## Root cause
The KMS allocates the per-team-wallet nonce (`obtain_app_id`) with an unlocked read-modify-write, so concurrent creates derive the same app address and collide. The proper server-side fix is tracked upstream in **Phala-Network/phala-cloud-monorepo#1544** (route `obtain_app_id` through the existing `.with_for_update()` lock).
## Fix we can ship here (no wait on the backend)
The provider already exposes `custom_app_id` + `nonce` on `phala_app` (verified on 0.3.0-beta.4), and the KMS has a predict endpoint `GET /kms/phala/next_app_id?counts=N`. So we can pre-assign distinct nonces and deploy in parallel:
1. Pre-flight: one serialized call to `next_app_id` for the team to get N distinct `{app_id, nonce}` pairs.
2. Feed one pair into each `phala_app` (coordinator + each workload group) via `custom_app_id` + `nonce`.
3. Drop the `-parallelism=1` requirement from the docs / README.
Each create then takes the `validate_manual_nonce` path with a pre-assigned distinct address, so the unique constraint is never contended.
### Constraints to handle
- `next_app_id` caps at 20 per batch; batch for larger fleets.
- Nonce must stay within ~20 of the team's current max; each nonce is single-use.
- Wiring the predict call into Terraform needs an `http`/external data source or a small pre-step script.
## Acceptance criteria
- `clusters/patroni-demo` deploys cleanly with `-parallelism > 1`, and two clusters apply concurrently without the nonce collision.
- README / docs no longer instruct `-parallelism=1`.
## Notes
- This is a client-side mitigation; once #1544 lands, the manual pre-allocation becomes optional (the auto path is safe again).
- Verified current behavior on provider `0.3.0-beta.4`, real cloud (h4xusers workspace).
Contributor guide
Assessment
This issue has not been assessed yet.