Agent Understanding of PointCloud2

The premise is that, given a correct string representation of a PointCloud2 object, an LLM could do spatial reasoning.

Video of an agent finding a nearby room, reporting its size, and assuming that objects nearby are furniture. It does not use the exploration skill. All target locations are manually picked by the agent after inspecting the pointcloud data. Note: captions generated by AI.

demo-2026-08-21-17-06-39.mp4

Timeline

PR 3415 introduces PointCloud2.agent_encode() and the testing methodology

PR 3415 introduced two things:

1. A PointCloud2's string representation meant for consumption by an agent: PointCloud2.agent_encode() -> str
2. 40 single-turn question/answer evals that followed this formula: <question that could be answered from a depth pointcloud> <encoded_pointclouds>

The evals fell under these families of questions:

• What are the extents (x, y, z) of this area?
• How far is the nearest obstacle?
• Given these multiple point clouds, did the total mapped area grow, shrink, or not change?
• Given these multiple point clouds, which cardinal direction was most explored?
• What is the total area in meters of the point cloud?
• Given these multiple point clouds, how far did the center shift?

Output

Given this scene:

agent_encode() returns this text:

{
    "frame_id": "world",
    "num_points": 23851,
    "centroid_xy_m": [
        -0.45,
        6.5
    ],
    "exact_stats": {
        "note": "exact full-cloud values in meters; for numeric extent/span/area answers use these, not the body-height interval map below",
        "x_range": [
            -3.47,
            2.83
        ],
        "y_range": [
            3.78,
            9.77
        ],
        "z_range": [
            -0.22,
            1.17
        ],
        "horizontal_extent_m": 6.3,
        "vertical_span_m": 1.4,
        "occupied_floor_footprint_m2": 37.4,
        "footprint_note": "mapped floor area of this frame's cloud only (distinct occupied 0.2 m x-y cells); use it, not bbox area, for floor coverage. For area trends compare each frame's own footprint value across frames -- it can decrease as well as increase; do not accumulate coverage over frames"
    },
    "compass": "8-way direction of motion (dx,dy = last minus first): if |dx|>2.41*|dy| then east (dx>0) or west (dx<0); if |dy|>2.41*|dx| then north (dy>0) or south (dy<0); otherwise diagonal by signs (northeast, northwest, southeast, southwest). For any question about which direction the map moved or gained coverage, take dx,dy from centroid_xy_m of the last minus the first frame; range edges are too noisy for direction",
    "body_height_occupancy": {
        "desc": "Exact bounding boxes (world meters) of obstacle points at body height (z 0.15..1.0 m), listed north to south. Each box xmin:xmax@ymin:ymax is the exact extent of its points (+x east, +y north; a lone value = zero-width, a thin obstacle). Horizontal clearance from a query point (qx,qy) = min over all boxes of hypot(dx,dy), where dx = max(0, xmin-qx, qx-xmax) and dy = max(0, ymin-qy, qy-ymax) (each term is zero only when the query lies inside that extent).",
        "boxes": "-2.17:-0.73@9.57:9.77,-3.38:-3.22@9.32:9.52,-1.83:-1.77@9.52,-1.42:0.32@9.32:9.52,1.23:1.27@9.52,1.67:1.77@9.38:9.48,2.22:2.33@9.32:9.43,-3.47:-3.28@9.07:9.27,0.08:0.32@9.27,0.82@9.18:9.23,-3.47@8.68,-3.17@8.68,-3.47:-3.08@8.38:8.43,-3.28@8.07,-1.33@8.07,-0.93@8.12,-3.33:-3.17@7.78:8.02,-2.03:-1.58@7.78:8.02,-1.27@7.93:7.97,1.02:1.08@7.78:7.82,-3.47:-3.28@7.53:7.68,-1.98:-0.88@7.53:7.72,0.62:1.88@7.53:7.72,-3.47:-3.22@7.38:7.47,-2.03:-1.33@7.28:7.47,-1.02:-0.88@7.28:7.47,0.62:1.48@7.28:7.47,1.77:1.83@7.32:7.38,-3.33:-3.28@7.18:7.22,-2.08:-0.93@7.03:7.22,0.62:1.92@7.03:7.22,-3.28@6.78,-2.08:-0.98@6.78:6.97,0.73:1.77@6.78:6.97,-2.12:-1.08@6.53:6.72,0.73:1.77@6.53:6.72,-3.47:-3.42@6.28:6.43,-2.53:-1.12@6.28:6.47,0.68:0.88@6.28:6.47,1.17:1.77@6.28:6.47,-3.47@6.22,-2.47:-1.12@6.03:6.22,0.68:1.73@6.03:6.22,-2.67:-1.12@5.78:5.97,0.28:0.38@5.78:5.93,0.68:0.88@5.78:5.97,1.23:1.73@5.78:5.93,-2.28:-1.27@5.53:5.72,0.22:0.77@5.53:5.72,1.23:1.73@5.53:5.72,0.73:0.77@5.38:5.43,1.17:1.67@5.28:5.47"
    }
}

where if we were to visualize the boxes defined inbody_height_occupancy, it would look like this:

Result

This resulted in > %90 success rate on the 40 evals.

However, agent_encode bakes in some assumptions about the caller:

• Perspective is fixed: It's akin to an orthographic top-down rendering
• Robot height is fixed: For a point to be considered an obstacle, it MUST be between 0.15<→1m in height
• Resolution is (sort of) fixed: Points are clustered into boxes or put into separate boxes based on the overall extents of the point cloud
• Boxes can't be angled

So, while agent_encode is a function on the PointCloud2 object, it's not really a general "encoding" of it - it's more-so doing the role of something like:

return select(my_cool_pointcloud)
    .where(height(max_step_altitude, robot_height))
    .2Dify()
    .AABB(resolution)

But what we undoubtedly learn is that an LLM is great at understanding reasoning about the stringified JSON representation handed to it!

Expansion of Eval Families

I introduce more families of questions:

• Is the area in this direction clear?
• Can you get to this location, which may or may not be accessible via a non-linear path?
• Can you fit in this area? (this was specifically testing for spotty lidar returns due to glass)
• How many rooms are there?
• How level is the floor? Are there stairs?
• Where are doorways?

Curating a dataset to create these evals was a lengthy procedure and is something that could be streamlined in the future. My process was as such:

• Record a session of driving the go2 around the SF office
• ask claude code to generate questions for the above families and provide visual proof corresponding to the coordinates it is claiming/asking about
• iterate on correctness so that (1) the answers were actually factually correct, (2) the questions were framed as real-world questions (i.e. not prescribing the agent how to think in the prompt), (3) the premise of the question was actually useful

Queue autoresearch.

Output

{
    ...original_values,
    "enclosure_topology": {
        "separate_areas": 2,
        "areas": "0.80,6.50@14.2;1.10,2.10@8.4",
        "openings": "0.95,4.30@0.90"
    },
    "ground_relief": {
        "floor_z": -0.05,
        "areas": [
            {
                "x": 2.3,
                "y": 3.1,
                "area_m2": 4.5,
                "dz_m": 0.18,
                "far_dz_m": 0.18
            }
        ]
    },
    "wall_openings": {
        "openings": [
            {
                "x": 0.95,
                "y": 4.3,
                "votes": 4
            }
        ]
    },
    "unmeasured_pockets": {
        "patches": "3.00:3.50@2.00:2.25,3.00:3.25@2.25:2.50"
    }
}

Results

The numbers were good but it was achieved by adding extra purpose-built post-processing of the PointCloud2 data in the agent_encode function - basically, more assumptions about the caller! Each new family of questions essentially brought about a new property added to the returned object.

The success rate of the new evals:

• Is the area in this direction clear? 96%→91%
• Can you get to this location, which may or may not be accessible via a non-linear path? 64%→80%
• Existence of glass: 40%→50% I am considering this set of evals as junk
• How many rooms are there? 0%→83%
• How level is the floor? 50%→94%
• Are there stairs? 50%→100%
• Where are doorways? 50%→95%

When actually demoing this in real-time, the agent stopped relying on its own reasoning capabilities over data and instead deferred to the pre-computed answers returned fromagent_encode. When I asked it "hey bud, how many rooms are there?" it just returned the answer found in enclosure_topology.separate_areas instead of looking at geometry primitives and intuiting from there.

If we can leverage that latter pathway - reasoning over lego blocks - then we for free benefit from the ever increasing reasoning abilities of future foundational models.

A Move to Generic Representation

First, I added more families of questions:

• which direction has the furthest range of unoccupied space?
• where is an area with the most free space? (i.e. a radius)
• is this object, who's immediate surrounding area has not been mapped, potentially reachable?

At this point there are 218 evals, with 70 of those being reserved as "holdouts" - evals that are only tested after autoresearch is done as a way to detect overfitting.

I then ran autoresearch with guidelines to remove instructions from agent_encode's response as well as not have properties that serve narrow usecases.

TL;DR: Output had something akin to a 2D heightmap and performance tanked for:

• cardinal directions
• floor level, stairs
• doors, rooms

Real-time Testing of the Agent

I would spin up either uv run dimos humancli or claude code with the DimOS MCP and give it some variation of these instructions:

# Premise
You are driving the connected DimOS robot via the mcp tools. We are evaluating your ability to understand your environment by interpreting pointcloud data (specifically its agent_encode function). During the course of this conversation, do not answer my questions by using tools that allow you to query the rgb camera or semantic labeling, but rather use the python tool and query the lidar data in memory2 - do not do a bunch of manual printing of pointcloud data. You are allowed to use the python tool to query your pose/odom information to get a grounding of yourself relative to the lidar info. You are expected to rely on agent_encode, not handwriting a bunch of math to iterate over raw pointcloud data.

# Navigating your environment
If you need to move, use the relative_move tool. Have your movements take up a meaningful amount of distance while staying approx 2m from obstacles. The lidar data you have is a small subset of the actual space as you've only mapped your immediate surroundings. You should look at your most recent pose from the odometry data to correlate your position and heading to the lidar data, and use that to determine where to move.

Testing an agent with only Python at runtime and no agent_encode

When I kicked off an autoresearch process, I wanted to explore the opposite end of things - boot up CC, connect it to the DimOS MCP, and tell it to navigate its environment/detect stairs/detect rooms just by writing python scripts over the mem2 stream at runtime.

The conclusions it would come to were generally not bad (it could find stairs! It could go to both ends of the stairs, it could go to specific x/y coords and explore an environment without using the exploration skill!), but it would literally sit there for five minutes doing twenty different numpy evaluations of the pointcloud data when I asked it to metaphorically take a breath. So there was a latency problem.

Allowing an agent to convert pointcloud data to actual images and view them

This was a brief side quest that I couldn't put the amount of time to it that it might deserve, but it had decent success with pointcloud interpretability via literally rendering depth maps from it. There is a lot of optimization to be done on the kinds of visualizations it renders and in general it had the same latency problem of doing extensive querying before taking any action.

Next Steps

The highest performance with lowest latency came from a prescriptive agent_encode. However, I believe the PointCloud2.agent_encode API will break down when we want an agent to use it for anything more than "where is there free space".

An agent was also able to be more adaptive and answer more nuanced spatial questions/tasks when doing raw operations against pointcloud data in python, but it would simply spend too much time doing so.

I want to test extracting some of the prescriptive behavior from agent_encode into an agent skill ("skill" meaning an actual agent harness skill, not @skill decorator that we use) and see if I can reduce the latency for the agent reasoning/action cycle.

I also think its fine to expose the occupancy grid as its own queryable primitive. It essentially does the heavy lifting of what agent_encode is doing, and it inherently holds semantic meaning for self-directed agent trajectories.

---

Synced from DIM-1509 by henry

Start with PointCloud2.agent_encode() and the 218-eval methodology described in the issue. Run the real-time workflow with uv run dimos humancli or inspect pointcloud data through the mem2 stream, then compare generic representations against the listed question families and holdouts. Done means improving general spatial reasoning without adding narrow, purpose-built properties or caller-specific assumptions.