Hardware h264 encoder fails to mux Matroska and separately fails Tee outputs in GStreamer and FFmpeg

Start by reproducing the listed FFmpeg h264_v4l2m2m and GStreamer v4l2h264enc commands on the Raspberry Pi CM4 environment. Compare hardware and software encoding for Matroska muxing and tee outputs, then inspect the reported encoder failures and timestamp errors. Done means hardware encoding works for Matroska output and both FFmpeg and GStreamer tee examples complete successfully.

Describe the bug

These may be two unrelated issues, but I have a hunch they may also be related to #1836 or #4734.

On my Compute Module 4 running 64bit Bullseye, I am unable to mux Matroska files when using the hardware h264 encoder. Further, I am also unable to use the tee functionality of both GStreamer and FFmpeg when using the hardware h264 encoder. Below are results from various tests:

Working - Hardware Encode

ffmpeg -y -f lavfi -i smptebars=duration=1:size=1920x1080:rate=30 -c:v h264_v4l2m2m out.h264

Possibly Benign Errors:
    Failed to set gop size: Invalid argument
    Encoder did not produce proper pts, making some up.
    Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1 >= 1

ffmpeg -y -f lavfi -i smptebars=duration=1:size=1920x1080:rate=30 -c:v h264_v4l2m2m out.mp4
ffmpeg -y -f lavfi -i smptebars=duration=1:size=1920x1080:rate=30 -c:v h264_v4l2m2m out.flv
ffmpeg -y -f lavfi -i smptebars=duration=1:size=1920x1080:rate=30 -c:v h264_v4l2m2m out.ts

Possibly Benign Errors:
    Failed to set gop size: Invalid argument
    Timestamps are unset in a packet for stream 0. 
    This is deprecated and will stop working in the future. 
    Fix your code to set the timestamps properly
    Encoder did not produce proper pts, making some up.

Broken - Hardware Encode

ffmpeg -y -f lavfi -i smptebars=duration=3:size=1920x1080:rate=30 -c:v h264_v4l2m2m out.mkv

Possibly Benign Errors:
    Failed to set gop size: Invalid argument

Breaking Errors:
    Could not write header for output file #0 (incorrect codec parameters ?):
    Invalid data found when processing input
    Error initializing output stream 0:0 --

gst-launch-1.0 -v videotestsrc pattern=snow num-buffers=100 ! v4l2h264enc \
! h264parse ! matroskamux ! filesink location=out.mkv

Breaking Errors::
    ../sys/v4l2/gstv4l2videoenc.c(828): gst_v4l2_video_enc_handle_frame (): 
    /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0:
    Maybe be due to not enough memory or failing driver
    ERROR: pipeline doesn't want to preroll.
    Setting pipeline to NULL ...
    ERROR: from element /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0: 
    Internal data stream error.

Working - Software Encode

ffmpeg -y -f lavfi -i smptebars=duration=3:size=1920x1080:rate=30 -c:v libx264 out.mkv

gst-launch-1.0 -v videotestsrc pattern=snow num-buffers=100 ! x264enc \
! h264parse ! matroskamux ! filesink location=out.mkv

Working Tee - Software Encode

ffmpeg -y -f lavfi -i smptebars=duration=1:size=1920x1080:rate=30 \
-c:v libx264 -f tee -map 0:v "out1.mp4|out2.mp4"

gst-launch-1.0 -v videotestsrc pattern=snow num-buffers=100 \
! video/x-raw,width=640,height=480 ! videoconvert \
! x264enc ! h264parse ! mp4mux ! tee name=t \
t. ! filesink async=0 location=out1.mp4 \
t. ! filesink async=0 location=out2.mp4

Broken Tee - Hardware Encode

ffmpeg -y -f lavfi -i smptebars=duration=1:size=1920x1080:rate=30 \
-c:v h264_v4l2m2m -f tee -map 0:v "out1.mp4|out2.mp4" 

Possibly Benign Errors:
    Failed to set gop size: Invalid argument
    Timestamps are unset in a packet for stream 0. 
    This is deprecated and will stop working in the future. 
    Fix your code to set the timestamps properly
    Encoder did not produce proper pts, making some up.

Breaking Errors:
    Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1 >= 1
    av_interleaved_write_frame(): Invalid argument

gst-launch-1.0 -v videotestsrc pattern=snow num-buffers=100 \
! video/x-raw,width=640,height=480 ! videoconvert \
! v4l2h264enc ! h264parse ! mp4mux ! tee name=t \
t. ! filesink async=0 location=out1.mp4 \
t. ! filesink async=0 location=out2.mp4

Breaking Errors:
    ERROR: from element /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0: 
    Failed to process frame.
    Additional debug info:
    ../sys/v4l2/gstv4l2videoenc.c(828): gst_v4l2_video_enc_handle_frame (): 
    /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0:
    Maybe be due to not enough memory or failing driver
    Execution ended after 0:00:00.021926718
    Setting pipeline to NULL ...
    ERROR: from element /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0: 
    Internal data stream error.
    Additional debug info:
    ../libs/gst/base/gstbasesrc.c(3127): gst_base_src_loop (): 
    /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0:
    streaming stopped, reason error (-5)

Steps to reproduce the behaviour

See the example commands above.

Device (s)

Raspberry Pi CM4

System

Raspberry Pi reference 2021-10-30
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, c12b1df4ed6416fb0df33ba1731c5b13c1bdbdf8, stage2

Jan 20 2022 13:56:48
Copyright (c) 2012 Broadcom
version bd88f66f8952d34e4e0613a85c7a6d3da49e13e2 (clean) (release) (start)

Linux PanoCamX 5.10.92-v8+ #1514 SMP PREEMPT Mon Jan 17 17:39:38 GMT 2022 aarch64 GNU/Linux

Logs

No response

Additional context

Please let me know if there's any particular debug / tests that would be helpful.