bluerobotics / bluerobotics/BlueOS

bug: mavlink-camera-manager loses frames on 1.5.0-beta.36

Open
#3,927 11 comments 0 reactions 1 assignee Claimed by @joaoantoniocardoso View on GitHub
bug triage ui
Dominant language
Vue
Stars
453
Forks
151
Avg merge
1d 3h
Merged PRs (30d)
174

Description

### Bug description

On BlueOS `1.5.0-beta.36`, `mavlink-camera-manager` appears to lose or delay H.264 frames/keyframes in the onboard video pipeline.

The issue causes visible multi-second glitches/gaps in the video stream. It is visible in the BlueOS stream thumbnails, Cockpit, Foxglove/MCAP playback, and onboard `.mcap` recordings, which suggests the problem originates before topside transport/display.

The same vehicle, camera setup, and topside setup work correctly after reverting to BlueOS `1.4.3`. The issue occurs both during dives and in dry bench tests. Changing stream resolution or frame rate did not resolve it.

In my test setup, I used two H.264 camera streams: the standard BlueROV camera and an ExploreHD USB camera. Observed frame rates were lower than configured:
- ExploreHD USB: configured 30 FPS, observed ~18.5 FPS
- H.264 USB stream: configured 15 FPS, observed ~11.6 FPS

The root cause is unclear. It may be related to CPU load or SD-card I/O, but I have not confirmed that.

### Steps to reproduce

1. Use a BlueOS system running `1.5.0-beta.36` with an H.264 camera stream.
2. Start the camera stream through `mavlink-camera-manager`.
3. Observe the stream in BlueOS stream thumbnails, in Cockpit on topside computer or recover it from onboard `.mcap` recorder file, if vehicle was armed during tests.
4. Note visible frame gaps/glitches or delayed recovery after missing keyframes/IDR frames.
5. Revert the same hardware/configuration to BlueOS `1.4.3` and repeat the test.
6. The stream works correctly on `1.4.3` with the same setup.

### Primary pain point(s)

_No response_

### Additional context

Only related issue I found is: https://github.com/bluerobotics/BlueOS/issues/3841

BlueOS PRs show related recent MCM/streaming changes: [`1.5.0-beta.36`](https://github.com/bluerobotics/BlueOS/releases/tag/1.5.0-beta.36) includes “Update mavlink camera manager”, and [`1.5.0-beta.34`](https://github.com/bluerobotics/BlueOS/releases/tag/1.5.0-beta.34) updated MCM to t3.23.2, MCAP extractor, recorder, and Zenoh-related logging.

During robot testing, BlueOS `1.5.0-beta.36` showed a much higher 15-minute load average than `1.4.3`.

A Raspberry Pi without vehicle hardware stayed around `~0.40` on both versions. On the BlueROV2 with two cameras, `1.4.3` was around `~1.40`, while `1.5.0-beta.36` exceeded `4`.

Several MCAP-related processes were active, including `mcap doctor` on multi-GB files. However, stopping `mcap doctor` and `mcap recover` did not noticeably improve the camera stream issue.

### Prerequisites

- [x] I have checked to make sure that a similar request has not already been filed or fixed.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.