Proposal: privileged ground-truth channel for interactive eval scoring (sim GT poses → evals framework)

Summary

The evals framework landed in #3411 scores InteractiveEval by sampling the live memory2 Store — i.e. state the runtime/agent itself produced. As @TomCC7 noted in review, that conflates the agent's perception with the world: scoring should be able to use privileged simulator ground truth (object poses, robot pose, contacts). This issue proposes a minimal channel for that, with a working prototype and measured evidence for why agent-side state is insufficient.

Motivating evidence (measured, not hypothetical)

While evaluating an xArm7 pick in the MuJoCo sim, we instrumented MujocoSimModule to publish free-joint body ground truth (details below) and ran the agent on "pick up the cup":

• The pick skill returned Pick complete — grasped 'sphere' successfully.
• Ground truth showed the approach/retract knocked all three scene objects off the table: orange z 0.175→0.045, cup z 0.190→0.035, apple z 0.170→0.040.

From memory2 / skill results alone this case scores as a success; against ground truth it is a catastrophic failure. Agent-side scoring also cannot express "was the cup lifted" at all when perception misses the object (our agent concluded "no cup is present in the scene" while the cup sat untouched). Related sim/planning consistency issue filed separately as #3484.

Proposal

Two small, separable pieces:

1. Sim modules publish ground truth on a privileged channel (opt-in).

MujocoSimModule already owns MjData; publishing free-joint scene bodies costs ~40 lines. Prototype shape we validated:

• Resolve free-joint bodies at start (exclude the robot root, already published as odom).
• In the post-step hook, publish each body's world pose as PoseStamped with frame_id = body name on a dedicated stream (e.g. /sim/gt/object_poses), throttled (we used 20 Hz; per-step would mirror odom).
• Behind a config flag (e.g. publish_ground_truth: bool = False) so production/real-robot setups are unaffected.

Prototype ran 700 s+ stable on zenoh, 3 objects @ 20 Hz, negligible overhead.

2. The evals framework lets scorers read that channel.

InteractiveEval.score currently receives only the agent's Store. Options, in order of our preference:

• a) EvalRig gains a gt_store() / GT accessor, fed by a Recorder subscribed to the privileged GT streams — scorer signature becomes score(store, gt), keeping today's score(store) as sugar;
• b) GT streams are simply additional mem2 streams in a privileged namespace, and selectors do the rest (no new interface, but mixes agent-writable and sim-only data in one store);
• c) scorer-side ad-hoc subscription (what our prototype does) — fine for experiments, too loose for suites.

Happy to open a PR for (1) + (2a) if the direction sounds right.

Notes

• This is sim-only privileged data; nothing here weakens the "mem2 is the source of truth for agent input" rule — GT is for scoring, never fed back to the agent.
• Grasp-relevant physics (contacts, held-object coupling) could come later on the same channel; object poses already cover lift/place/knock-over scoring.

Investigated and drafted with AI assistance (Kimi Code CLI); the prototype and all measurements were run on the referenced setup.

Start by reading MujocoSimModule, InteractiveEval.score, EvalRig, and the Recorder path described in the proposal. Compare the three GT-access options and confirm how opt-in simulator poses remain separate from agent input. Done means an agreed channel and scorer interface support ground-truth evaluation without affecting production or real-robot setups.