MiniCPM-o 4.5 video-only LoRA on Hugging Face Jobs: compatibility notes and workarounds

Hi, I ran a small video-only LoRA pilot with openbmb/MiniCPM-o-4_5 using LLaMA-Factory and Hugging Face Jobs.

This is not yet a minimal bug reproduction. I wanted to share the compatibility issues and workarounds I encountered, since they may help other users trying to fine-tune MiniCPM-o 4.5 for video-text SFT without audio.

Setup

• Model: openbmb/MiniCPM-o-4_5
• Model revision: 4382fcae8a551b54d18f18462db974ff312aa7f3
• Training framework: LLaMA-Factory
• LLaMA-Factory revision: 0b7aaf8f6a624bd89a01a155d4265ec861cbdf38
• Cloud runtime: Hugging Face Jobs, A100
• Task: video-only soccer commentary generation
• Data shape: 30-second video clips, 448p, 4 fps, no audio
• LoRA target modules: q_proj,v_proj
• Vision tower frozen: yes
• Multimodal projector frozen: yes

Issues Encountered

1. Video-only training still triggered audio-related initialization paths.
2. LLaMA-Factory forward arguments conflicted with MiniCPM-o remote code in this setup.
3. An in-place visual embedding scatter operation caused autograd compatibility issues during training.
4. For inference, transformers==4.56.2 exposed a DynamicCache.seen_tokens compatibility issue, while pinning inference to transformers==4.51.0 worked in my run.
5. Continuing LoRA training from a previous adapter required explicitly downloading the PEFT adapter revision locally; otherwise the adapter revision and base model revision could be mixed, causing adapter_config.json lookup failure.

Workarounds Used

For training, I applied a local compatibility patch that:

• gated audio embedding calls when running video-only training;
• removed duplicate forward kwargs before calling the underlying LLM;
• replaced in-place visual embedding scatter_ with an autograd-safe non-in-place path;
• patched LLaMA-Factory's MiniCPM-o config path to use init_audio=False for video-only training.

For inference, I used:

• transformers==4.51.0
• init_vision=True
• init_audio=False
• init_tts=False
• generate_audio=False
• enable_thinking=False

Result

After these workarounds, the pipeline completed successfully. I trained a small LoRA adapter on 300 video-text samples and evaluated it on 100 frozen test clips.

This issue is mainly to ask:

1. Is video-only LoRA training for MiniCPM-o 4.5 officially supported through LLaMA-Factory?
2. Is init_audio=False the recommended path for video-only SFT?
3. Would you prefer a minimal reproduction issue, a documentation note, or a PR for this compatibility path?

I am happy to prepare a minimal reproduction or documentation PR if this direction is useful.

Thanks for the great model and open-source work.

Start by reproducing the video-only LoRA setup with the listed MiniCPM-o, LLaMA-Factory, and Transformers revisions, focusing on the init_audio flags, forward arguments, visual embedding operation, and adapter loading. Done means determining whether video-only SFT is supported, confirming the required compatibility settings, and documenting an accepted workaround or minimal reproduction.