fix(orchestrator): envd error responses truncated to 100 chars, hiding root cause in logs and error chain
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1.6k
- Forks
- 438
- PR merge metrics
- No merged PRs in 30d
Description
Problem
In packages/orchestrator/pkg/sandbox/envd.go, all envd error-response bodies are truncated to 100 characters via utils.Truncate(string(body), 100) before being embedded in fmt.Errorf strings and log fields:
```go
// line 182 — callEnvdCollapse
fmt.Errorf("collapse returned %d: %s", resp.StatusCode, utils.Truncate(string(body), 100))
// line 208 — postEnvd (covers /init, /pause, /resume, …)
fmt.Errorf("%s returned %d: %s", path, resp.StatusCode, utils.Truncate(string(body), 100))
// line 287 — CallEnvdUpgrade
fmt.Errorf("upgrade returned %d: %s", resp.StatusCode, utils.Truncate(string(body), 100))
// line 496 — initEnvd log field
zap.String("response_body", utils.Truncate(string(body), 100))
```
Because lines 182, 208, and 287 inject the truncated string directly into the error value, the truncation propagates all the way up the error chain and surfaces to callers as:
```
failed to create sandbox: failed to wait for sandbox start: failed to init new envd: /init returned 400: failed to mount "7e22c38a-5584-4730-8c31-449f5058ea19": failed to chroot into "/data1/orchestrator/vo
```
The message is cut mid-path, losing the actual root cause.
Impact
NFS volume mount failures are a concrete example. The full error body from envd is ~300 characters:
```
failed to mount "7e22c38a-5584-4730-8c31-449f5058ea19": failed to chroot into "/data1/orchestrator/volumes/team-f0796066-fb0b-4ea7-a9fe-613e9b8831bb/vol-885b64ff-686d-4f5c-b16d-eb4d5f9cd577": failed to bind mount "/data1/orchestrator/volumes/team-f0796066-fb0b-4ea7-a9fe-613e9b8831bb/vol-885b64ff-686d-4f5c-b16d-eb4d5f9cd577": no such file or directory
```
At 100 characters the message is truncated to:
```
failed to mount "7e22c38a-5584-4730-8c31-449f5058ea19": failed to chroot into "/data1/orchestrator/vo
```
The operator cannot tell from orchestrator logs or the returned error whether the failure is a missing directory, a permission issue, an NFS timeout, or something else. Diagnosing the root cause requires cross-referencing a completely separate service's logs (the NFS proxy).
Fix
Remove utils.Truncate from the four error-path call sites. envd error response bodies are short diagnostic strings generated by the server, not arbitrary user data, so there is no log-explosion risk.
PR: #3481
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read packages/orchestrator/pkg/sandbox/envd.go and inspect the four envd error and logging call sites identified in the issue. Verify how response bodies flow into returned errors and logs, then confirm the completed change preserves the full diagnostic body at each site without truncation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, infrastructure
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100