dimensionalOS / dimensionalOS/dimos
Investigate Agentic understanding of space
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.5k
- Forks
- 808
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 233
Description
if memory2 query returns an occupancygrid (or pointcloud?) how can we present it to a LLM model in a way in which it understands it?
write an eval, ask basic questions like
spatial understanding
model will guess - so we need a lot of very specific questions
"how many rooms there are in total"
"how big is that room down the hallway to the right?"
"where is the biggest room in this space"
"what type of space is this"
point placement
"where should we explore the space next?"
"place a point on a biggest/smallest room"
"place points marking hallways"
some (failed) attempt here of just encoding occupancygrid as a picture for VLM https://github.com/dimensionalOS/dimos/pull/822
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the failed occupancy-grid-as-picture attempt in PR #822. Define an evaluation for presenting occupancy grids or point clouds to an LLM, covering specific spatial-understanding and point-placement questions; done means the evaluation can measure answers to those questions.
Written by the indexing model from the issue text.
Assessment
- Domain
- ai, robotics
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100