client-proxy 502 errors & orchestration-api snapshot not found: poor observability and ambiguous error messages

Summary

In production environments, we are observing two critical issues that lead to sandbox startup failures, ambiguous user-facing errors, and poor debuggability:

1. client-proxy floods with 502 Reverse proxy errors

   • No upstream health checking
   • No TCP/grpc connectivity validation before forwarding
   • Returns generic 502 instead of meaningful 503 Service Unavailable
   • Causes sandbox process startup to fail intermittently

2. orchestration-api returns misleading 404 on snapshot not found

   • Log: snapshot not found
   • User error: Sandbox doesn't exist or you don't have access to it
   • No detailed logging for why snapshot lookup failed (missing object, path, permissions, cache)
   • Service still registers routes as ready despite being unable to serve snapshots

These issues make it extremely hard to debug sandbox startup failures and provide a bad user experience.

Log Evidence

client-proxy 502
Reverse proxy error {"service": "client-proxy", "shturl.cc/": "im2pa8g5739aezjy57vr1", "target_hostname": "10.254.73.19", "target_port": "5007", "status_code": 502}

orchestration-api snapshot not found
snapshot not found {"service": "orchestration-api", "shturl.cc/": "im2pa8g5739aezjy57vr1"}

Start by tracing the client-proxy reverse-proxy path and the orchestration-api snapshot lookup using the supplied log examples. Compare the observed 502 and misleading 404 behavior with the requested service-availability and diagnostic outcomes, including readiness when snapshots cannot be served. Done means startup failures expose actionable errors and the relevant failures are logged with enough context to distinguish their causes.