perception - aruco detector should emit Detection3D

MarkerTfModule (dimos/perception/fiducial/marker_tf_module.py) bypasses the detection type system — estimates pose with solvePnP and publishes Transform straight to TF as marker_{id}. no Detection object ever exists.

a marker is a Detection3D in every sense: 3D pose, source image, known size, class (dictionary + id), image-space bbox from the corners. all of that is currently thrown away.

if it emits Detection3D we get for free:

• ImageDetections.filter(...) pipeline
• annotated_image() debug overlay
• consistent treatment with yolo/sam — spatial memory, object tracking
• a reusable to_rerun() story (see "wire format" below)

TF publishing stays, but as a downstream consumer of the detection stream, not the source.

type shape

Detection3DBBox already fits: center, size=(L, L, 0) for planar, orientation, frame_id, transform, plus bbox from corners' image-space rect, class_id=marker_id, name=f"{dict}:{id}".

if marker-specific fields are worth modeling → Detection3DMarker(Detection3DBBox) adds dictionary: str, corners_2d: list[tuple[float, float]], reprojection_error: float.

no track_id — marker_id IS the persistent identity. probably stuffed into class_id (already str in the ROS msg).

module shape — first pure-3D detector

Detection3DModule (dimos/perception/detection/module3D.py:45) is really "2D detector → unproject via pointcloud" — subclasses Detection2DModule, requires pointcloud: In[PointCloud2], calls Detection3DPC.from_2d(...).

a marker module is the first pure-3D detector: pose comes straight from solvePnP, no 2D-then-project hop, no pointcloud. doesn't fit under Detection3DModule as-is.

options:

• new MarkerDetectionModule(Module) sibling — takes color_image + camera_info, emits Detection3DArray directly. doesn't subclass Detection3DModule.
• refactor Detection3DModule into an abstract base that just promises detections: Out[Detection3DArray]; both "2D+pointcloud" and "pure 3D" inherit. probably overkill until there's a third one.

(side: Detection3DModule.detections is Out[Detection2DArray] — should be Detection3DArray. adjacent, not this issue.)

wire format — making this visualizable

We send ROS msgs across the wire, those are not our custom Detection types.

rerun bridge (dimos/visualization/rerun/bridge.py:256) subscribes to pubsubs and calls to_rerun() on whatever lands on the wire

In case above this will be the ROS msg (Detection3DArray), not the Python Detection3DBBox. so to_rerun() on the Python class alone wouldn't render anything, we need .to_rerun on ros Detection3DArray

what survives Detection3DBBox.to_detection3d_msg(): ts, frame_id, class_id (str), confidence, center, orientation, size. dropped: source image, image-space bbox, marker fields.

Equivalent Foxglove features for reference

https://github.com/dimensionalOS/dimos/pull/859 — initial detection annotations
https://github.com/dimensionalOS/dimos/pull/1017 — segmentation rendering (note: shouldn't render both bbox + mask, just mask)

wire format -- reconstruction (this is not important for this feature, just informing in general)

Sometimes other module needs our rich Detection3DMarker type on the other side
this is done (in case of 2d detections) by associating ros image msg to ros detection msgs

Detection3DArray + Image matched by timestamp

this pattern already exists for 2D:

• Detection2DBBox.from_ros_detection2d(ros_det, image=image) — dimos/perception/detection/type/detection2d/bbox.py:282
• ImageDetections2D.from_ros_detection2d_array(image, ros_array) — dimos/perception/detection/type/detection2d/imageDetections2D.py:38
• consumers pair image + detection streams via align_timestamped: dimos/perception/detection/reid/module.py:62, dimos/perception/detection/person_tracker.py:92

open questions

• Detection3DMarker subclass, or stuff into Detection3DBBox (class_id=marker_id, name=dict)?
• sibling module, or refactor Detection3DModule into an abstract base first?
• just embed in class_id string like "DICT_APRILTAG_36h11:42"?

---

Synced from DIM-739 by summer