feat(agents): agent-queryable memory

Problem

The robot continuously builds a rich spatial record via memory2 (camera frames, odometry, lidar) -- every observation is marked with a world pose and timestamp. However, at this point, the McpClient has no access to any of it at runtime; it can only reason about conversation history.

This means the agent cannot inherently answer questions that require spatial memory, e.g:

1. Walk towards the brightest part of the room (after exploring)
2. Who did you greet at the door earlier?
3. Have you been here before?

Proposed High-level Solution

Approach: wrap memory2's existing store API in @skill methods so the agent discovers them through the standard get_skills() handshake, introduce a dedicated AgentMemory module for episodic notes, and update the system prompt to encourage proactive use -- no changes to McpClient or the skill registration pipeline.

1. _query_store() on MemoryModule

Applies spatial (.near()), temporal (.after()), and semantic filters (embedding-based search on modules with a model) in any combination. All params are optional. Semantic search is silently skipped on modules without an embedding model.

  def _query_store(self, stream: str, semantic_query: str | None = None,
                   near_x: float | None = None, near_y: float | None = None,
                   radius: float = 2.0, after_seconds_ago: float | None = None,
                   k: int = 5) -> str:
      s = self.store.stream(stream)
      if near_x is not None and near_y is not None:
          s = s.near((near_x, near_y), radius)
      if after_seconds_ago is not None:
          s = s.after(time.time() - after_seconds_ago)
      if semantic_query and getattr(self, "model", None) is not None:
          s = s.search(self.model.embed_text(semantic_query), k=k)
      results = s.limit(k).to_list()
      if not results:
          return "No matching observations found."
      return "\n".join(   
          f"t={datetime.fromtimestamp(obs.ts).strftime('%H:%M:%S')} "
          f"pose={obs.pose_tuple} "
          f"tags={obs.tags}" 
          + (f" data={obs.data:.120}" if isinstance(obs.data, str) else "")
          for obs in results
      )

2. Explicit @skill on Go2Memory (or other memory modules)

Each recorder that wants agent access defines a @skill that delegates to _query_store(), plus a companion list_*_streams skill for stream discovery. The existing get_skills() / on_system_modules path picks both up with zero changes.

class Go2Memory(Recorder):
    color_image: In[Image]
    lidar: In[PointCloud2]
    odom: In[PoseStamped]
    config: Go2MemoryConfig

    @skill
    def query_go2memory(self, stream: str, semantic_query: str | None = None,
                        near_x: float | None = None, near_y: float | None = None,
                        radius: float = 2.0, after_seconds_ago: float | None = None,
                        k: int = 5) -> str:
        """Query Go2 robot memory. Call list_go2memory_streams() to see available streams.
        semantic_query: natural language (embedding-based, only on streams with a model).
        near_x/near_y: world XY to filter by proximity. after_seconds_ago: recency filter."""
        
        return self._query_store(stream, semantic_query, near_x, near_y, radius, after_seconds_ago, k)

    @skill
    def list_go2memory_streams(self) -> str:
        """List memory streams available to query on the Go2 robot."""
        
        return ", ".join(self.store.list_streams()) or "No streams recorded yet."

3. Dedicated AgentMemory(Recorder) module

Rather than placing remember() on MemoryModule (where all Recorder subclasses inherit it), introduce a dedicated AgentMemory(Recorder) module that owns remember() and the "events" stream. This keeps sensor memory (Go2Memory: what the robot perceived) cleanly separated from episodic memory (AgentMemory: what the agent experienced and decided). Drops into any blueprint as a standalone module via the standard get_skills() handshake.

  class AgentMemoryConfig(RecorderConfig):
      db_path: str | Path = "agent_memory.db"

  
  class AgentMemory(Recorder):
      """Episodic memory for the agent — stores what the agent experienced and decided.

      Separate from sensor recorders (Go2Memory) which store what the robot perceived.
      Exposes remember() and query_agent_memory() as skills, discovered automatically
      via get_skills().
      """

      config: AgentMemoryConfig
      model: EmbeddingModel | None = None

      @rpc
      def start(self) -> None:
          super().start()
          self.model = self.register_disposable(CLIPModel())                                                                                                                                                               
          self.model.start()

      @skill
      def remember(self, note: str) -> str:
          """Store an episodic note for future recall. Call after meaningful interactions —
          meeting someone, completing a task, observing something noteworthy."""
          events = self.store.stream("events", str)
          pose = self.tf.get("world", "base_link")
          embedding = self.model.embed_text(note)
          events.append(note, ts=time.time(), pose=pose.to_pose() if pose else None,
                        embedding=embedding)
          return "Stored."

      @skill
      def query_agent_memory(self, semantic_query: str | None = None,
                             near_x: float | None = None, near_y: float | None = None,
                             radius: float = 2.0, after_seconds_ago: float | None = None,
                             k: int = 5) -> str:
          """Query episodic agent memory. Supports semantic, spatial, and temporal filters."""                                                                                                                             
          return self._query_store("events", semantic_query, near_x, near_y,
                                   radius, after_seconds_ago, k)

4. Memory section in system_prompt.py

A brief system prompt instruction is needed to encourage the LLM to use its memory context. This may require some fine-tuning as LLMs don't always reach for tools like this proactively:

  ## Memory
  
  You have persistent spatial memory. Use it.

  - To recall sensor data (what you saw, where you were), call
    `list_go2memory_streams()` then `query_go2memory()` with a relevant query.
  - To recall past interactions and decisions, call `query_agent_memory()`.
  - After meaningful interactions — meeting someone, completing a task, observing
    something noteworthy — call `remember()` with a concise note written for future
    recall. Example: `remember("Met Bob near kitchen entrance. He asked for water.")`
  - Do not call `remember()` for routine actions like navigation steps or sensor checks.

5. McpClient -- no changes needed out of the box

Memory skills register through the same get_skills() / on_system_modules handshake as every other skill. McpClient discovers query_go2memory, list_go2memory_streams, and remember at startup and wraps them as tools identically to any other tool. The existing HTTP → McpServer → LCM RPC dispatch path handles them without modification.

If memory use turns out to be too passive in practice, a follow-on option is deterministic injection -- proactively calling a memory tool at the start of each turn before the LLM responds.

Open questions

Spatial param usability near_x/near_y require world-frame coordinates. To solve for this, we would need to lean on two different methods:

1. Historical location (tool chaining): query_go2memory results already include pose_tuple on every observation. The LLM can query a memory which returns a location, then re-use those XY coordinates in a follow-up spatial query. This requires the LLM to reliably parse and re-use coordinates from a result string. (e.g. "Go to the kitchen from earlier")
2. Per-turn pose injection: McpClient inherits self.tf from ModuleBase and prepends [Current pose: x=..., y=..., heading=...] to each HumanMessage in _process_message. This gives the LLM world coordinates for free every turn. (this would help with "Is there anyone in the area around me right now?")

remember() trigger policy. Should notes be written at the LLM's discretion, deterministically after every turn, or triggered when conversation history approaches the context limit? LLM-driven is cleanest but may underfire; deterministic is reliable but noisy and expensive.

Dynamic stream names in skill descriptions. Stream names are currently surfaced via list_go2memory_streams(), which requires the LLM to make a separate call before calling query_go2memory An optimization would be a callable description= param on @skill evaluated at get_skills() time against the live instance so descriptions always reflect actual streaming ports.

Start by tracing MemoryModule's existing store API and the get_skills()/on_system_modules path, then inspect the Go2Memory and Recorder entry points plus system_prompt.py. Done means sensor and episodic memory skills are discoverable at runtime, support the proposed queries, and the prompt guides appropriate memory use; resolve the listed spatial and remember() policy questions before implementation.