dimensionalOS / dimensionalOS/dimos
Complete manipulation behavior tree
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.5k
- Forks
- 808
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 233
Description
Summary
Integrate the behavior tree-based pick-and-place pipeline with GraspGen to enable robust, agentic pick-and-place capability in dimOS.
Context
There are two relevant pieces that need to come together:
- Behavior tree pick-and-place — A
py_trees-based orchestration layer exists on thejalaj_BT_pickplacebranch (dimos/manipulation/bt/). It has async execution, retry/recovery logic, and grasp verification — but it's out of sync with the currentmainbranch and needs rework. - GraspGen — A Docker-based deep-learning grasp generation module already on main (
dimos/manipulation/grasping/graspgen_module.py). It's production-ready and supports collision filtering. - Heuristic pick-and-place — The current
PickAndPlaceModuleon main (dimos/manipulation/pick_and_place_module.py) works end-to-end but uses simple heuristic grasps and has no retry/recovery logic.
The goal is to combine the robustness of the BT approach with the grasp quality of GraspGen, wired up to the agentic interface.
Key pointers
| What | Where |
|---|---|
| Current heuristic pick-and-place (main) | dimos/manipulation/pick_and_place_module.py |
| BT pick-and-place (stale branch) | jalaj_BT_pickplace branch — dimos/manipulation/bt/ |
| GraspGen module | dimos/manipulation/grasping/graspgen_module.py |
| Manipulation blueprints | dimos/manipulation/blueprints.py |
| Agentic blueprint with skills | xarm_perception_agent in dimos/manipulation/blueprints.py |
| Base manipulation module | dimos/manipulation/manipulation_module.py |
Deliverables
- Rebase/rework the BT pick-and-place module onto current
main - Integrate GraspGen as the primary grasp source in the BT pipeline (with heuristic fallback)
- End-to-end pick-and-place working in the
xarm_perception_agentblueprint - Agentic interface — agent can invoke
pick,place,pick_and_placeskills and receive success/failure feedback - Basic test coverage for the BT tree structure
Notes
- The BT branch was removed from main due to CI failures — start by understanding what broke before merging code back.
- The existing
PickAndPlaceModuleskills API (pick,place,drop_on,scan_objects, etc.) is the target interface the agent already knows how to use — preserve it. - GraspGen runs in Docker and is lazy-initialized on first call. The BT branch already has a
GenerateGraspsaction node that calls it.
Synced from DIM-747 by stash
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing dimos/manipulation/pick_and_place_module.py, dimos/manipulation/grasping/graspgen_module.py, dimos/manipulation/blueprints.py, and dimos/manipulation/manipulation_module.py with the dimos/manipulation/bt/ code on the jalaj_BT_pickplace branch. First understand the CI failures that removed the branch, then trace the existing pick, place, and pick_and_place skills in xarm_perception_agent. Done means the BT pipeline works with the existing agentic interface, GraspGen fallback behavior, and basic tree-structure coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- machine-learning, robotics
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100