[HOW-TO] Investigate loss of frames

Start with the provided RpiCamera code and the timestamp-analysis script, then compare the 50 FPS and 30 FPS runs, including the reported frame and loss counts. Check the encoder start/stop sequence and timestamp output first; done means identifying a reproducible cause for the stutter or documenting why the timestamp results do not match playback.

Hello,

I am developping a program that allows to both visualize the video feed in real time in a web browser and allow to manually trigger recordings.

Here is the code I use

import io
import threading

from libcamera import controls, Transform

from picamera2 import Picamera2
from picamera2.encoders import MJPEGEncoder, H264Encoder, Quality
from picamera2.outputs import FileOutput

class StreamingOutput(io.BufferedIOBase):
    def __init__(self):
        self.frame = None
        self.condition = threading.Condition()

    def write(self, buf):
        with self.condition:
            self.frame = buf
            self.condition.notify_all()

class RpiCamera():

    def __init__(self):
        # Declare picam
        self.picam2 = Picamera2()

        self.is_recording = False

        # Create and set video configuration
        self.video_config = self.picam2.create_video_configuration(main={"size": (1920, 1080)},
                                                    lores={"size": (640, 360)},
                                                    controls={
                                                        'FrameRate': 50,
                                                        'NoiseReductionMode': 1,
                                                        'AfMode': controls.AfModeEnum.Continuous},
                                                    transform=Transform(hflip=0, vflip=0))

        self.picam2.configure(self.video_config)

        # Instanciate MJPEGEncoder used to serve the mjpeg stream
        self.mjpegEncoder = MJPEGEncoder()
        # Streaming output
        self.streaming_output = StreamingOutput()
        # Set encoder outputs
        self.mjpegEncoder.output = [FileOutput(self.streaming_output)]
        self.picam2.start_encoder(self.mjpegEncoder, name="lores")

        # Instanciate H264Encoder
        self.h264encoder = H264Encoder(framerate=50, enable_sps_framerate=True)

        self.picam2.start()

    def start_recording(self):
        if not self.is_recording:
            self.picam2.start_recording(self.h264encoder, "test.h264", quality=Quality.HIGH, pts="test-timestamps.txt")
            self.is_recording = True
            return True
        return False

    def stop_recording(self):
        if self.is_recording:
            # I do not use stop_recording as it will stop every encoders
            self.picam2.stop_encoder(self.h264encoder)
            self.is_recording = False
            # Instanciate a new H264Encoder in order to reset the timestamps
            self.h264encoder = H264Encoder(framerate=50, enable_sps_framerate=True)
            return True
        return False

    # Method that generates the frames to be displayed (not used in the example)
    def gen(self):
        while True:
            with self.streaming_output.condition:
                self.streaming_output.condition.wait()
                frame = self.streaming_output.frame
            yield (b'--frame\r\n'
                   b'Content-Type: image/jpeg\r\n\r\n' + frame + b'\r\n\r\n')

if __name__ == '__main__':

    rpi_camera = RpiCamera()

    while True:
        user_input = input("Enter '1' to start recording, '0' to stop recording, or 'quit' to exit the program: ").lower()

        if user_input == '1':
            rpi_camera.start_recording()
        elif user_input == '0':
            rpi_camera.stop_recording()
        elif user_input == 'quit':
            print("Exiting program...")
            break
        else:
            print("Invalid input. Please enter 'start', 'stop', or 'quit'.")

When checking the video stream on a webpage (not implemented in this example) for some reason the video stream begins to stutter after a few recordings, and when checking the recorded h264 file, it sometimes stutters for a while before resuming after a few seconds, resulting in lost frames, and other times there are so much lost frames that the video playback is quite choppy, like if it was recorded at 10 FPS or so.

I tried to investigate using this script by @davidplowman :

import sys

with open(sys.argv[1], 'r') as f:
    timestamps = [float(line.rstrip()) for line in f.readlines()]
diffs = [round((next - now) / 20.0) - 1 for now, next in zip(timestamps, timestamps[1:])]
print("Total frames:", len(timestamps), " lost:", int(sum(diffs)))

Running this gives me: Total frames: 838 lost: 1571, but I am not sure how to interpret the results.

When setting the parameters to 30 FPS instead of 50, I do not seem to reproduce the issue, but running the same script as above returns Total frames: 17970 lost: 18019 but this doesn't seem right as the video plays back nicely.

I don't know where to begin to look in order to fix this issue, can someone point me in the right direction ?

I thank you in advance.

Regards,

Azsde.