esphome / esphome/feature-requests

Improve ESP32 camera frame rates when streaming video to HA

Open
#2,653 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
No language data
Stars
450
Forks
29
PR merge metrics
No merged PRs in 30d

Description

**Describe the problem you have/What new integration you would like**

Current frame rates with 640x480 of an OV2640 are > 10 FPS when streaming with MJPEG web camera component, but 2.5 FPS when watching the same feed through HA. Rates remain high when both videos are playing, however when the MJPEG component ceases to stream, rates drop to 2.5 FPS.

Similar effect is seen with the OV5640, where the MJPEG streaming rate is 6.5 FPS average, and drops to 1.5 when streaming through HA API.

(These tests were performed using a Freenove device, although other ESP devices were tested with similar results. Freenove needed a custom component with minimal edits compared to the upstream component: https://github.com/Rudd-O/esphome_freenove_camera_component )

**Please describe your use case for this integration and alternatives you've tried:**

A higher frame rate would enable much more uses of OV cameras with ESP32 devices.

**Additional context**

No additional context. I do have this code to calculate the frame rate:

```yaml

camera:
# ...
on_image:
- component.update: camera_frame_rate

globals:
- id: last_update_times
type: unsigned int[$frame_rate_buffer_size]
restore_value: no

sensor:
- platform: template
name: "Camera frame rate"
id: camera_frame_rate
lambda: |-
float now = millis();
unsigned int max_ = $frame_rate_buffer_size;
unsigned int i;
for (i = max_ - 1; i > 0; i--) {
id(last_update_times)[i] = id(last_update_times)[i-1];
}
id(last_update_times)[0] = now;
float avgdiff = 0.0;
float mostrecentmillis = id(last_update_times)[0];
for (i = 1; i < max_; i++) {
if (mostrecentmillis == 0) {
ESP_LOGD("Camera", "No more updates, breaking at %u", i);
break;
}
avgdiff = mostrecentmillis - id(last_update_times)[i];
if ((id(last_update_times)[i-1] - id(last_update_times)[i]) > 9500) {
ESP_LOGD("Camera", "Big diff, breaking at %u", i);
if (i > 1) {
// This is at least value number three from most
// recent to least recent. Undo its use.
i = i - 1;
avgdiff = mostrecentmillis - id(last_update_times)[i];
ESP_LOGD("Camera", "Compensating avgdiff");
}
break;
}
}
if (avgdiff == 0.0) {
return {};
} else {
avgdiff = avgdiff / i;
float rate = 1000.0 / avgdiff;
//if (rate >= 5.0) {
// rate = floor(rate + 0.5);
//} else {
// rate = floor(rate * 10 + 0.5) / 10;
//}
return rate;
}
update_interval: never
entity_category: diagnostic
state_class: measurement
device_class: frequency
filters:
- heartbeat: 5s
- throttle_average: 5s
- delta: 0.25
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by comparing the ESP32 MJPEG web camera component's frame rate with the feed delivered through the Home Assistant API, using the OV2640 and OV5640 measurements as baselines. Review the referenced Freenove custom component and reproduce the streaming conditions. Done means achieving a higher, consistent Home Assistant frame rate without degrading the MJPEG stream.

Written by the indexing model from the issue text.

Assessment

Domain
audio-video-rtc, embedded-iot
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.