Proposal: Persistent Entity Tracking over Memory2 Streams

Motivation

Memory2 provides Store, Stream, transform, align, and semantic search capabilities. For example, an image Stream can be transformed into CLIP embeddings and queried using text.

However, semantic similarity alone does not determine whether observations from different timestamps belong to the same physical object. Searching for a cup may return multiple frames of the same cup without providing a deduplicated movement history.

We would like to explore an object-centric pipeline over Memory2 Streams:

• Stage 1 produces stable 2D track observations from image Streams.
• Stage 2 adds point-cloud evidence and associates 2D and 3D tracks into entity observations.
• Semantic queries are then performed over representative frames associated with these entities.

Current DimOS components

Based on the current DimOS code:

• Yolo2DDetector uses YOLO11 and Ultralytics tracking. Its effective default tracker is BoT-SORT.
• YoloPersonDetector explicitly uses BoT-SORT.
• Yoloe2DDetector provides YOLOE detection, segmentation and short-term tracking.
• ReidModule provides an appearance-embedding-based identity system, but its long-term identity result is not currently emitted as a Memory2 Stream.
• Detection3DModule, ProjectDepthTo3D and ProjectTo3D can lift 2D detections into 3D using depth or a point cloud.
• The current demos do not provide an independent pipeline that performs instance segmentation and multi-object tracking directly from Stream[PointCloud2].

BoT-SORT is not intrinsically a human detector: it tracks detections supplied by the detector. Its original benchmarks and some current DimOS demos focus on people, while this proposal aims to extend persistent tracking to general objects such as cups, bags, tools and other everyday items. This will also require evaluating ReID representations suitable for general objects rather than relying only on person-oriented ReID models.

Stage 1: 2D Detection, Tracking and ReID

Stage 1 processes Stream[Image] and produces observations grouped by a short-term track_id.

The initial experiment will:

1. Connect the existing YOLO11 and BoT-SORT path to Memory2 replay.
2. Aggregate ImageDetections2D by track_id.
3. Compare BoT-SORT and ByteTrack using the same detector and replay data.
4. Evaluate whether the existing ReidModule can provide useful identity evidence for general objects.
5. Add YOLO26 as another detector candidate and compare it with the current YOLO11 baseline.

The expected output is:

Stream[Image]
→ 2D Detection
→ 2D Tracking
→ Track Aggregation
→ Appearance ReID
→ Stream[2D Track Observations]

Stage 1 does not assume that a local track_id is already a permanent physical identity. Instead, it produces the tracklets and identity evidence required by the next stage.

Stage 2: Point-cloud Processing and Entity Association

DimOS already contains PointCloud2, TF lookup and 2D-to-3D projection components. However, the current demos primarily lift RGB or RGB-D detections into 3D; they do not yet provide point-cloud-native instance segmentation followed by 3D multi-object tracking.

For the first macOS experiment, we propose starting with a lightweight Open3D pipeline—for example, point-cloud filtering and DBSCAN clustering—followed by a simple temporal association baseline. Open3D already provides the relevant point-cloud operations and can run without CUDA.Open3D point-cloud clustering

We can also evaluate recent research as follow-up candidates:

• Any3DIS (CVPR 2025) performs class-agnostic 3D instance segmentation using tracked 2D masks and multi-view 3D evidence. It is architecturally relevant, although its code and license need to be reviewed before integration.
• GRAE-3DMOT (CVPR 2025) focuses on 3D tracking-by-detection and geometric association. Its current setup requires CUDA and its repository license is not yet complete, so it is more suitable for a later NVIDIA experiment than the initial macOS implementation.

The initial Stage 2 integration will therefore prioritize a runnable and license-compatible baseline, while keeping the model boundary replaceable for later experiments.

Entity Alignment and Association

Time alignment, Pose/TF lookup and 2D–3D matching should be handled together as one entity-association pipeline rather than presented as unrelated steps.

Its inputs include:

Stream[2D Track Observations]
Stream[3D Track Observations]
appearance identity evidence
observation timestamps and Pose/TF

Its output is:

Stream[Entity Observations]

Conceptually, each entity history should support the following structure:

entity_id:
  timestamp:
    track_2d_id
    track_3d_id
    match_probability

This allows a query to recover not only the frames assigned to an entity, but also how its 2D and 3D tracks were associated at each timestamp and how confident that association was.

Trajectory descriptions may later be derived from these histories and supplied to an LLM/VLM for higher-level reasoning. Such reasoning would consume the structured tracking result rather than replace geometric and temporal association.

Proposed data pipeline

flowchart TB
    STORE["Memory2 Store"]

    STORE --> IMAGE["Stream[Image]"]
    STORE --> POINTS["Stream[PointCloud2]"]

    IMAGE --> DETECT2D["transform<br/>2D Detection"]
    DETECT2D --> TRACK2D["transform<br/>2D Tracking"]
    TRACK2D --> DETS2D["Stream[ImageDetections2D]"]

    DETS2D --> GROUP2D["transform<br/>Track Aggregation"]
    GROUP2D --> TRACKS2D["Stream[2D Track Observations]"]

    TRACKS2D --> REID["transform<br/>Appearance ReID"]
    REID --> IDENTITY["Stream[Identity Evidence]"]

    POINTS --> SEGMENT3D["transform<br/>3D Instance Segmentation"]
    SEGMENT3D --> TRACK3D["transform<br/>3D Tracking"]
    TRACK3D --> TRACKS3D["Stream[3D Track Observations]"]

    TRACKS2D --> ASSOC
    IDENTITY --> ASSOC
    TRACKS3D --> ASSOC

    ASSOC["Entity Association Pipeline<br/>time alignment + Pose/TF + probability"]
    ASSOC --> ENTITIES["Stream[Entity Observations]"]

    classDef existing fill:#F3F4F6,stroke:#4B5563,stroke-width:2px,color:#111827;
    classDef stage1 fill:#DBEAFE,stroke:#2563EB,stroke-width:2px,stroke-dasharray:6 4,color:#1E3A8A;
    classDef stage2 fill:#FFEDD5,stroke:#EA580C,stroke-width:2px,stroke-dasharray:6 4,color:#7C2D12;

    class STORE,IMAGE,POINTS existing;
    class DETECT2D,TRACK2D,DETS2D,GROUP2D,TRACKS2D,REID,IDENTITY stage1;
    class SEGMENT3D,TRACK3D,TRACKS3D,ASSOC,ENTITIES stage2;

Semantic query

After entity observations are available, representative frames can be embedded once and queried by text:

flowchart TB
    subgraph PREPROCESS["Preprocessing"]
        direction LR
        ENTITY["entity_id<br/>+ entity observations"]
        FRAME["Representative frames"]
        IMAGE_EMBED["CLIP image embedding"]
        EMBEDDINGS["entity_embeddings"]

        ENTITY --> FRAME
        FRAME --> IMAGE_EMBED
        IMAGE_EMBED --> EMBEDDINGS
    end

    subgraph QUERY["Query"]
        direction LR
        TEXT["Text"]
        TEXT_EMBED["CLIP text embedding"]
        SEARCH["Search entity_embeddings"]
        MATCHED_ID["entity_id"]
        HISTORY["Retrieve entity history"]

        TEXT --> TEXT_EMBED
        TEXT_EMBED --> SEARCH
        SEARCH --> MATCHED_ID
        MATCHED_ID --> HISTORY
    end

    EMBEDDINGS --> SEARCH

Feature goals and dataset request

A longer-term goal is to handle objects outside person-tracking benchmarks, including objects with similar appearance, long occlusion and re-entry.

One useful stress test is the shell game: several visually similar cups move, cross and become occluded, while the system maintains a probability over their physical identities and containment relations.

If the DimOS team can help record synchronized RGB, LiDAR and Pose/TF data with physical-object identity ground truth, it would allow Stage 1 and Stage 2 to be evaluated on the same controlled sequences.