MiniCPM-V 4.6 as a VLM backbone in StarVLA for LIBERO robot manipulation
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 26.4k
- Forks
- 2.1k
- Avg merge
- 14h 39m
- Merged PRs (30d)
- 3
Description
Hi MiniCPM-V team,
I wanted to share a downstream integration / community use case of MiniCPM-V 4.6 in embodied AI.
We recently integrated MiniCPM-V 4.6 as a new vision-language backbone for StarVLA, a Vision-Language-Action framework for robot manipulation.
StarVLA PR: https://github.com/starVLA/starVLA/pull/354
The integration adds:
- A MiniCPM-V 4.6 VLM wrapper for StarVLA
- PI / GR00T-style VLA framework support
- LIBERO training and evaluation scripts
- Example configs for 8-GPU training
Using MiniCPM-V 4.6, we trained and evaluated on the LIBERO benchmark and got the following results:
| Benchmark | Success Rate |
|---|---|
| LIBERO-Spatial | 94.0% |
| LIBERO-Object | 98.0% |
| LIBERO-Goal | 98.0% |
| LIBERO-10 | 92.4% |
| Overall | 95.6% |
Training setup:
- Backbone:
openbmb/MiniCPM-V-4.6 - Effective batch size: 128
- Max training steps: 80k
- Attention implementation:
flash_attention_2 - No modules are frozen by default (
FREEZE_MODULES=""), so MiniCPM-V is trainable unless overridden.
MiniCPM-V 4.6 looks like a promising lightweight VLM backbone for embodied AI / VLA-style robot learning. The initial LIBERO results are encouraging, especially given the compact model size compared with many larger VLM backbones.
As a next step, we are also considering testing this MiniCPM-V-based StarVLA model on a real SO-101 robot setup, to further evaluate its sim-to-real potential beyond LIBERO simulation benchmarks.
Thanks for releasing MiniCPM-V!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
This issue reports a downstream StarVLA integration rather than naming a change in MiniCPM-V. Read the linked StarVLA PR and the reported LIBERO training setup first; completion criteria for work in this repository are not specified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, robotics
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100