integrate with LeRobot
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 589
- Forks
- 76
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 3
Description
Is your feature request related to a problem? Please describe.
Currently rai-manipulation-demo uses ros2 moveit and a fusion of segmentation with depth camera to control Panda robotic arm.
This approach is good to for reasoning and planning, but might be limited when it comes to more complex movements of the gripper. For such task, policies implemented in LeRobot can be more useful.
LeRobot currently offers:
- 4 pre-trained policies listed in Models section: link
- scripts for policies training: link
- dataset format for training policies: link
Describe the solution you'd like
I find the integration of all LeRobot features mentioned above useful in rai, but I think a step-by-step should be applied.
- First logical step would be to attach pre-trained policy to the rai agent as a tool.
- Next would be to determine what policies would be useful for manipulation agent and to train them on datasets curated in simulation.
- Next step would be to generate and record trajectories for training using rai and text descriptions
Describe alternatives you've considered
Additional context
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading docs/demos/manipulation.md and the linked LeRobot policy, training, and dataset documentation. The issue proposes several integration stages but names no implementation entry point, tests, or acceptance criteria; clarify whether the first deliverable is attaching a pre-trained policy as an agent tool and define how completion will be verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, machine-learning, robotics
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100