OpenBMB / OpenBMB/MiniCPM-V

MiniCPM-V 4.6 as a VLM backbone in StarVLA for LIBERO robot manipulation

Open
#1,109 2 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
26.4k
Forks
2.1k
Avg merge
14h 39m
Merged PRs (30d)
3

Description

Hi MiniCPM-V team,

I wanted to share a downstream integration / community use case of MiniCPM-V 4.6 in embodied AI.

We recently integrated MiniCPM-V 4.6 as a new vision-language backbone for StarVLA, a Vision-Language-Action framework for robot manipulation.

StarVLA PR: https://github.com/starVLA/starVLA/pull/354

The integration adds:

  • A MiniCPM-V 4.6 VLM wrapper for StarVLA
  • PI / GR00T-style VLA framework support
  • LIBERO training and evaluation scripts
  • Example configs for 8-GPU training

Using MiniCPM-V 4.6, we trained and evaluated on the LIBERO benchmark and got the following results:

Benchmark Success Rate
LIBERO-Spatial 94.0%
LIBERO-Object 98.0%
LIBERO-Goal 98.0%
LIBERO-10 92.4%
Overall 95.6%

Training setup:

  • Backbone: openbmb/MiniCPM-V-4.6
  • Effective batch size: 128
  • Max training steps: 80k
  • Attention implementation: flash_attention_2
  • No modules are frozen by default (FREEZE_MODULES=""), so MiniCPM-V is trainable unless overridden.

MiniCPM-V 4.6 looks like a promising lightweight VLM backbone for embodied AI / VLA-style robot learning. The initial LIBERO results are encouraging, especially given the compact model size compared with many larger VLM backbones.

As a next step, we are also considering testing this MiniCPM-V-based StarVLA model on a real SO-101 robot setup, to further evaluate its sim-to-real potential beyond LIBERO simulation benchmarks.

Thanks for releasing MiniCPM-V!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

This issue reports a downstream StarVLA integration rather than naming a change in MiniCPM-V. Read the linked StarVLA PR and the reported LIBERO training setup first; completion criteria for work in this repository are not specified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, robotics
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.