[Bug]: applyCloudRelayConfig is not atomic; a failed secret write after the relay link is committed leaves the environment stuck on endpoint_provider_not_managed
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 23k
- Forks
- 5.9k
- Avg merge
- 11h 14m
- Merged PRs (30d)
- 357
Description
Before submitting
- I searched existing issues and did not find a duplicate.
- I included enough detail to reproduce or investigate the problem.
Area
apps/server
Steps to reproduce
Context: this is the root cause behind the endpoint_provider_not_managed connection failures in #6568. Filing it separately because it is one specific defect with a specific fix, and the other issue has grown to cover several symptoms.
What the relink flow does today
linkPrimaryEnvironmentToCloud (apps/web/src/cloud/linkEnvironment.ts) runs three steps in order:
POST /v1/client/environment-link-challengeson the relayPOST /v1/client/environment-linkson the relay. This upsertsrelay_environment_linksfor(userId, environmentId), includingendpointProviderKind(infra/relay/src/environments/EnvironmentLinks.ts,upsert). At this point the relay's copy of the link is already committed.POST /api/connect/relay-configon the local environment server, handled byapplyCloudRelayConfig(apps/server/src/cloud/http.ts). That handler writes six secrets one at a time throughServerSecretStore.set, in this order:cloud-relay-url,cloud-relay-issuer,cloud-linked-user-id,cloud-relay-environment-credential,cloud-mint-ed25519-public-key, thencloud-endpoint-runtime-config(set or removed depending on the mode).
There is no transaction and no rollback around step 3. ServerSecretStore.set is atomic per file (write tmp, rename), but the six writes together are not. If any write after the first fails, the relay has the new link and the local server keeps the previous credential, user id, mint key, and runtime config.
How to reproduce deterministically
- Link an environment normally in managed mode so
cloud-endpoint-runtime-config.binexists and cloudflared is running. - Make the second secret write fail. Any of these works: make
~/.t3/userdata/secrets/cloud-relay-issuer.binread-only, hold it open with a tool that takes an exclusive lock, or on Windows let an on-access antivirus scanner hold the file during the rename. (renameover an open file fails withEPERMon Windows; the store maps that toSecretStorePersistErrorand the handler returns "Could not persist environment relay configuration.") - In Settings > Connections, with Publish agent activity on, turn T3 Connect off. The controller relinks in
publish_onlymode (useCloudLinkController.ts,mode: desired.managedTunnel ? "managed" : "publish_only"), so step 2 storesendpointProviderKind = "manual"on the relay. - Step 3 fails after the first write. Local state is now:
cloud-relay-url.binrewritten, everything else from the old managed link, cloudflared still running against the old tunnel. - Try to connect from the mobile app.
Expected behavior
Either the whole relink applies or none of it does. Concretely, one of:
- Persist the local relay config before committing the relay-side link (obtain everything, write all secrets to staging, then upsert on the relay, then promote the staged secrets), or
- Keep the current order but roll back the relay link (re-upsert the previous endpoint, or unlink) when
relay-configfails, or - At minimum, make
applyCloudRelayConfigwrite all six secrets to temp files first and only rename them into place once every temp write has succeeded, so local state cannot end up half-updated.
Actual behavior
Relay and desktop disagree about the link and nothing reconciles them:
- Relay:
endpointProviderKind = "manual".EnvironmentConnector.resolveManagedEndpoint(infra/relay/src/environments/EnvironmentConnector.ts) rejects every connect and status call withendpoint_provider_not_managed. - Desktop:
readCloudLinkStatereportsmanagedTunnelActive: truebecausecloud-endpoint-runtime-configstill exists. cloudflared keeps the old tunnel up, so the environment stays discoverable and health checks pass, which is why the mobile app lists it and then fails on connect. - The Connections UI only relinks when
managedTunnelActive !== desired.managedTunnel, so turning the toggle on again is a no-op (that part is tracked separately, see the linked UI issue).
Real occurrence on my machine: on 2026-09-11 11:30:26 local, cloud-relay-url.bin was rewritten and the other five secrets kept their 2026-07-31 timestamps. The mobile app reported Connection failed. Reason: Relay rejected the environment connection request (endpoint_provider_not_managed). trace id: ad7746aff889c11de16074675a73acda from then until I forced a full relink on 2026-09-15.
Impact
Blocks work completely
Version or commit
Desktop 0.0.40 (release channel). Code references are against main as of 2026-09-15.
Environment
Windows 11 Home 10.0.26200, T3 Code desktop 0.0.40, iOS T3 Code app, McAfee real-time scanning enabled at the time of the failed write.
Logs or stack traces
# secrets dir after the half-applied relink (2026-09-11), everything else untouched since the original link
-rw-r--r-- 411 2026-07-31 23:49:58 cloud-endpoint-runtime-config.bin # still providerKind cloudflare_tunnel
-rw-r--r-- 32 2026-07-31 23:49:58 cloud-linked-user-id.bin
-rw-r--r-- 112 2026-07-31 23:49:58 cloud-mint-ed25519-public-key.bin
-rw-r--r-- 167 2026-07-31 23:49:58 cloud-relay-environment-credential.bin
-rw-r--r-- 22 2026-07-31 23:49:58 cloud-relay-issuer.bin
-rw-r--r-- 22 2026-09-11 11:30:26 cloud-relay-url.bin # only file the relink touched
# after a successful relink (2026-09-15 09:45:45) all six land within 10 ms of each other
-rw-r--r-- 411 2026-09-15 09:45:45.291 cloud-endpoint-runtime-config.bin
-rw-r--r-- 32 2026-09-15 09:45:45.285 cloud-linked-user-id.bin
-rw-r--r-- 112 2026-09-15 09:45:45.289 cloud-mint-ed25519-public-key.bin
-rw-r--r-- 167 2026-09-15 09:45:45.287 cloud-relay-environment-credential.bin
-rw-r--r-- 22 2026-09-15 09:45:45.283 cloud-relay-issuer.bin
-rw-r--r-- 22 2026-09-15 09:45:45.281 cloud-relay-url.bin
Workaround
Force a full relink so the relay record is rewritten as cloudflare_tunnel: Settings > Connections, turn Publish agent activity off, turn T3 Connect off (this is the only combination that actually unlinks), then turn T3 Connect on, then publish back on. Turning only the T3 Connect toggle off and on does not help while publishing is on, because "off" becomes a publish-only relink rather than an unlink.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with applyCloudRelayConfig in apps/server/src/cloud/http.ts and trace ServerSecretStore.set, then review linkPrimaryEnvironmentToCloud in apps/web/src/cloud/linkEnvironment.ts and the relay upsert in infra/relay/src/environments/EnvironmentLinks.ts. Reproduce a failed secret write and verify that a relink either updates every secret and the relay link or leaves both sides consistent without partial local state.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100