feat(observability): add correlated diagnostics for managed outbound transports
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
## Problem Statement
Applications inside a NemoClaw/OpenShell sandbox often see only a high-level transport error such as `fetch failed`, `request timed out`, or `disconnect/reset before headers`. That output does not identify whether the failure occurred during policy evaluation, proxy CONNECT, TLS setup, upstream connection, response headers, or an application protocol exchange.
This is especially difficult when the same service works through a direct connection but fails through the managed proxy. Operators currently have to correlate agent logs, CLI output, and OpenShell audit logs manually, often without a shared request identifier.
MCP is the first concrete consumer, but the diagnostic contract should be reusable by messaging, provider, webhook, package-fetch, and other managed outbound transports.
## Desired Behavior
Provide a reusable, failure-only managed-transport diagnostic facility that emits a redacted structured event and a correlation identifier across the application and OpenShell proxy boundaries.
For example:
```text
managed_transport_failure
consumer=mcp
operation=tools/list
route=trusted_env_proxy
proxy=:
target=:
phase=response_headers
http_status=503
elapsed_ms=1512
cause_code=UND_ERR_SOCKET
response_server=envoy
x_request_id=
x_envoy_response_flags=
trace_id=
```
The first integration should instrument OpenClaw remote Streamable HTTP MCP around its actual HTTP fetch boundary. A single failed probe should make it possible to determine:
- whether the managed proxy was used;
- which phase failed: policy, CONNECT, TLS, application connect, request, response headers, or response stream;
- the safe HTTP status and selected proxy/upstream response headers;
- the complete sanitized error-cause chain;
- elapsed time and transport/protocol mode;
- whether an opaque application session identifier was present, without exposing it;
- the matching OpenShell audit/proxy event through a shared trace identifier.
A contributor should be able to reuse the same event schema, redaction helpers, and phase classification for a different networked integration without copying an OpenClaw-specific formatter.
## Constraints and Non-goals
- Never log authorization headers, cookies, OAuth tokens, query strings, request bodies, client keys/certificates, or full application session identifiers.
- Error-body capture must be non-2xx-only, bounded, redacted, content-type-aware, and must not interfere with streaming response consumption.
- Normal successful traffic should not generate verbose per-request logs by default.
- Diagnostics must not bypass policy, alter proxy selection, weaken TLS verification, or automatically retry requests.
- The initial issue does not need a universal packet-capture system or a new tracing backend.
- Do not claim that a proxy-generated response proves whether the proxy itself or its upstream generated the reset; preserve evidence needed for that determination.
## Implementation Idea
1. Define a small shared managed-transport diagnostic schema and redaction helper under NemoClaw's observability/networking layer.
2. Add a version-scoped OpenClaw dist patch that wraps the remote MCP HTTP fetch boundary and emits the shared event only on failure.
3. Generate or propagate a safe request/trace identifier and include it in both application diagnostics and available OpenShell audit events.
4. Capture safe response metadata such as status, `server`, `via`, `x-request-id`, and `x-envoy-*` diagnostic fields when present.
5. Walk nested error causes and retain safe fields such as error name, code, errno, syscall, address family, and port.
6. Reuse the existing CONNECT/policy/timeout classifier vocabulary rather than creating a second incompatible taxonomy.
7. Add a focused CLI presentation that can collect a timestamp-bounded probe result plus matching sandbox logs without exposing secrets.
8. Test with deterministic fake proxies for policy denial, CONNECT failure, TLS failure, upstream reset, HTTP 503, timeout, and truncated/redacted error bodies.
## Acceptance Criteria
- [ ] One failed OpenClaw MCP probe produces a structured, redacted phase classification.
- [ ] Application and available OpenShell events share a correlation identifier.
- [ ] Proxy-generated status and safe diagnostic headers are retained when present.
- [ ] Nested transport causes are preserved without leaking credentials or request content.
- [ ] Streaming MCP responses remain behaviorally unchanged.
- [ ] The schema/helper is reusable by at least one non-MCP test fixture or example consumer.
- [ ] Tests prove redaction, body bounds, failure-only logging, and deterministic phase classification.
- [ ] Documentation explains how to collect and interpret the correlated evidence.
## Category
Platform
## Checklist
- [x] I searched existing issues and this is not a duplicate
- [x] I described the problem and desired behavior
Contributor guide
Research direction
Start at the OpenClaw remote Streamable HTTP MCP fetch boundary and review the existing CONNECT, policy, and timeout classifier vocabulary before defining shared schema and redaction behavior. Exercise the focused CLI and deterministic fake-proxy cases named in the issue; done means failed probes produce correlated redacted events without changing streaming behavior, with a reusable non-MCP fixture and documentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend-api-design, networking, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100