dimensionalOS / dimensionalOS/dimos

feat(protocol): per-RPC structured trace records for agent execution evidence

Open
#3,437 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.5k
Forks
808
Avg merge
3d 5h
Merged PRs (30d)
233

Description

Problem

DimOS has no structured record of individual RPC calls. #1499 added JSONL logging at the MCP tool-call boundary (tool name, args, truncated response, duration), but anything below that boundary is invisible: a module calling another module's @rpc, Dimos.connect() clients, and — most acutely — code authored by an agent inside a persistent policy kernel (#3378, previously #3259) all issue RPCs that leave no trace. When a multi-RPC policy rollout fails, neither the agent nor a human reading main.jsonl can tell which call failed, with what arguments, and after how long. Agent repair loops (ASPIRE-style) and eval diagnostics both need this per-call evidence.

Proposal

Add a per-RPC structured trace at the two transport-agnostic choke points in dimos/protocol/rpc/spec.py, so every implementation (Zenoh, LCM, SHM) is covered with no per-backend work:

  • Client side: RPCClient.call_sync / call_async
  • Server side: RPCServer.serve_module_rpc's dispatch wrapper

One rpc_trace structlog event per call, into the existing per-run main.jsonl:

{"event": "rpc_trace", "direction": "client", "rpc": "ManipulationModule/move_to_joints",
 "args": "(['0.3,-0.5,0.0,1.2,0.0,0.8,0.0'], {})", "status": "ok", "duration_ms": 1958.6}
{"event": "rpc_trace", "direction": "server", "rpc": "ManipulationModule/move_to_joints",
 "args": "...", "status": "error", "error": "SkillError", "duration_ms": 12.1}
  • statusok | error | timeout; error carries the exception class name.
  • args is a bounded repr (500 chars) since RPC payloads can carry images/point clouds.
  • Purely observational: no RPC behavior change, no new dependencies.

Verified end-to-end against a real Zenoh client/server pair — success, server-side exception (propagates to both sides), and timeout (client-only record) all produce the expected records.

Relationship to existing work (non-overlap)

  • #1499: MCP boundary logging. This proposal is the layer below — it also captures RPCs issued from inside python_exec programs, which never cross the MCP boundary per call.
  • #3339 (PimSim trace contracts): defines eval-gate contracts; this provides the runtime capture those contracts can consume.
  • #3410 (code policy authoring): records eval artifacts (policy source, transcripts); per-RPC runtime evidence is complementary and could feed its "execution evidence".
  • #2053 (LangSmith/Langfuse): LLM-call observability, orthogonal to RPC-level evidence.

Non-goals for v1

  • No cross-process trace-ID propagation (client and server records stand alone; correlation can come later).
  • No query API or agent-facing exposure yet — the records land in main.jsonl first; surfacing a machine-readable subset back to the agent is a follow-up.

Open questions

  • Always-on vs config-gated? #1499 set the precedent of always-on boundary logging; RPC volume inside a run is modest (high-frequency control loops use SHM streams, not RPC), but happy to add a GlobalConfig switch if preferred.
  • Truncation limit (500) and whether result values should also be recorded client-side (currently args only; results are often large).

A working implementation with tests is ready; I can open the PR as soon as this is triaged.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read dimos/protocol/rpc/spec.py, starting with RPCClient.call_sync, call_async, and the RPCServer.serve_module_rpc dispatch wrapper. Verify the existing per-run main.jsonl logging path and exercise a Zenoh client/server pair for success, server errors, and timeouts. Done means bounded rpc_trace records with the specified fields and statuses, without changing RPC behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design, distributed-systems, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.