Memory Store Spec

WIP - will write actual type specs soon

still specifiying this

Memory store

on every dimos run, automatically store all sensor data as binary blobs + metadata in appropriate format

optional post-run cleanup/deletion, settings for max space used etc like rerun has

efficient storage and indexing based on time/space/and embeddings

• pluggable backends for diff data (psql/sqlite/pickle implemented)
• sqlite vector extensions seem very good for multidim search/embeddings

Easily add new programatic overlays for temporal/spatial/semantic data

so we don't just store sensor streams, but maybe LLM thoughts, some sort of reprocessing, even debug data,
also be able to dynamically create a new reprocessed data streams, suddenly we care about bottles last 2 weeks, we can generate

replaces our save/replay system completely

you like your run? it's already stored, just name it, mark it as permanently available so it doesn't get auto cleaned up

run traditional search algorithms on top of this data

I can ask "give me a list of most bottle like things, within this physical space, with this kind of temporal bias"

when searching for a bottle, since we are doing a search in mutlidimensional space (time, embeddings, space as search heuristics) we can utilize traditional search algos, simulated anneling etc, heuristics can be pluggable

then I can (easily!) run VLM query on top, potentially output the same data type again - TemporalData augmented/biased with VLM, run more queries on top "give me latest with score above 0.7", give me a spatial heatmap to overlay on my occupancygrid, give me temporal graph of semantic similarity to this third query etc

corelate tempral stores

"give me lidar frame closest in time to this image frame" - so I can project something into space

The issue is explicitly WIP and names no files, tests, or entry points. Begin by documenting the type specs and reviewing the existing save/replay system plus the stated SQLite and other backend options; done should be an agreed, actionable memory-store specification covering storage, indexing, cleanup, overlays, search, and temporal correlation.