dimensionalOS / dimensionalOS/dimos

Complete manipulation behavior tree

Open
#1,670 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

good trial issue manipulation
Dominant language
Python
Stars
4.5k
Forks
808
Avg merge
3d 5h
Merged PRs (30d)
233

Description

Summary

Integrate the behavior tree-based pick-and-place pipeline with GraspGen to enable robust, agentic pick-and-place capability in dimOS.

Context

There are two relevant pieces that need to come together:

  1. Behavior tree pick-and-place — A py_trees-based orchestration layer exists on the jalaj_BT_pickplace branch (dimos/manipulation/bt/). It has async execution, retry/recovery logic, and grasp verification — but it's out of sync with the current main branch and needs rework.
  2. GraspGen — A Docker-based deep-learning grasp generation module already on main (dimos/manipulation/grasping/graspgen_module.py). It's production-ready and supports collision filtering.
  3. Heuristic pick-and-place — The current PickAndPlaceModule on main (dimos/manipulation/pick_and_place_module.py) works end-to-end but uses simple heuristic grasps and has no retry/recovery logic.

The goal is to combine the robustness of the BT approach with the grasp quality of GraspGen, wired up to the agentic interface.

Key pointers

What Where
Current heuristic pick-and-place (main) dimos/manipulation/pick_and_place_module.py
BT pick-and-place (stale branch) jalaj_BT_pickplace branch — dimos/manipulation/bt/
GraspGen module dimos/manipulation/grasping/graspgen_module.py
Manipulation blueprints dimos/manipulation/blueprints.py
Agentic blueprint with skills xarm_perception_agent in dimos/manipulation/blueprints.py
Base manipulation module dimos/manipulation/manipulation_module.py

Deliverables

  • Rebase/rework the BT pick-and-place module onto current main
  • Integrate GraspGen as the primary grasp source in the BT pipeline (with heuristic fallback)
  • End-to-end pick-and-place working in the xarm_perception_agent blueprint
  • Agentic interface — agent can invoke pick, place, pick_and_place skills and receive success/failure feedback
  • Basic test coverage for the BT tree structure

Notes

  • The BT branch was removed from main due to CI failures — start by understanding what broke before merging code back.
  • The existing PickAndPlaceModule skills API (pick, place, drop_on, scan_objects, etc.) is the target interface the agent already knows how to use — preserve it.
  • GraspGen runs in Docker and is lazy-initialized on first call. The BT branch already has a GenerateGrasps action node that calls it.

Synced from DIM-747 by stash

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing dimos/manipulation/pick_and_place_module.py, dimos/manipulation/grasping/graspgen_module.py, dimos/manipulation/blueprints.py, and dimos/manipulation/manipulation_module.py with the dimos/manipulation/bt/ code on the jalaj_BT_pickplace branch. First understand the CI failures that removed the branch, then trace the existing pick, place, and pick_and_place skills in xarm_perception_agent. Done means the BT pipeline works with the existing agentic interface, GraspGen fallback behavior, and basic tree-structure coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
machine-learning, robotics
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.