huggingface / huggingface/transformers

R3M: A Universal Visual Representation for Robot Manipulation

Open
#16,403 0 comments 0 reactions 0 assignees View on GitHub
New model
Dominant language
Python
Stars
166k
Forks
34.6k
Avg merge
3d 9h
Merged PRs (30d)
281

Description

# 🌟 New model addition

## Model description
We pre-train a visual representation using the Ego4D human video dataset using a combination of time-contrastive learning, video-language alignment,and an L1 penalty to encourage sparse and compact representations. The resulting representation, R3M, can be used as a frozen perception module for downstream policy learning. Across a suite of 12 simulated robot manipulation tasks, we find that R3M improves task success by over 20% compared to training from scratch and by over 10% compared to state-of-the-art visual representations like CLIP and MoCo. Furthermore, R3M enables a Franka Emika Panda arm to learn a range of manipulation tasks in a real, cluttered apartment given just 20 demonstrations.

## Open source status

* [x] the model implementation is available:(https://github.com/facebookresearch/r3m)
* [x] the model weights are available: https://github.com/facebookresearch/r3m/blob/main/r3m/example.py
* [x] who are the authors: @suraj-nair-1

Contributor guide

Open the contributing guide

Research direction

Start with the linked R3M implementation and weights in r3m/example.py, then review Transformers model-integration conventions. The issue names no Transformers entry point or tests, so the integration scope and validation still need to be established; done means R3M is supported using the available implementation and weights.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
computer-vision, machine-learning, robotics
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.