RobotecAI / RobotecAI/rai

integrate with LeRobot

Open
#345 14 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement help wanted priority/minor
Dominant language
Python
Stars
589
Forks
76
Avg merge
1d 13h
Merged PRs (30d)
3

Description

Is your feature request related to a problem? Please describe.
Currently rai-manipulation-demo uses ros2 moveit and a fusion of segmentation with depth camera to control Panda robotic arm.
This approach is good to for reasoning and planning, but might be limited when it comes to more complex movements of the gripper. For such task, policies implemented in LeRobot can be more useful.

LeRobot currently offers:

  • 4 pre-trained policies listed in Models section: link
  • scripts for policies training: link
  • dataset format for training policies: link

Describe the solution you'd like
I find the integration of all LeRobot features mentioned above useful in rai, but I think a step-by-step should be applied.

  • First logical step would be to attach pre-trained policy to the rai agent as a tool.
  • Next would be to determine what policies would be useful for manipulation agent and to train them on datasets curated in simulation.
  • Next step would be to generate and record trajectories for training using rai and text descriptions

Describe alternatives you've considered

Additional context

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading docs/demos/manipulation.md and the linked LeRobot policy, training, and dataset documentation. The issue proposes several integration stages but names no implementation entry point, tests, or acceptance criteria; clarify whether the first deliverable is attaching a pre-trained policy as an agent tool and define how completion will be verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, machine-learning, robotics
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.