Delayed H264 decoding on browsers when using VideoDecoder for macOS devices

I have been trying to put together an end to end H264 real-time streaming implementation from macOS devices to browsers. The macOS implementation uses native hardware accelerated APIs to encoder the frames, convert the encoded buffers from AVCC format to AnnexB format and send it across over the wire, on the other end the browser receives the stream and feeds it to the VideoDecoder WebCodecs API. I am using hardwareAcceleration: 'no-preference' setting on the VideoDecoder so the browser uses hardware accelerated decoder provided by the operating system.

The problem I am facing is the decoder takes about couple of seconds(around 3 seconds most of the times) to decode the frames, irrespective of the profile I use, I have tried with High, Main and Baseline. As I am trying to achieve real time streaming even a more than few hundreds of milli seconds overall is not acceptable. I did verify the SPS, PPS, I and P frames and they all look good to me. Below is the config I am using on the browser to create the decoder

const config = {
  codec: "avc1.64001F",
  avc: { format: "annexb" },
  codecWidth: 1440,
  codecHeight: 900,
  hardwareAcceleration: 'no-preference',
  optimizeForLatency: true
}

I am also attaching a encoded bitstream for your reference of 17 frames (15 P-Frames separated by 2 I-Frames), can anyone help me to understand what is going on with the decoder/encoder here?

data_0_0_2025-07-23 0:23:41.1310.dat.zip

PS: When I decoded the same encoded bitstream using native decoder API, I do not see any issues, I can play the saved decoded video without any issues. I see issue only with browsers.

Start by reproducing the delay with the attached encoded bitstream and the provided VideoDecoder configuration, including codec, Annex B format, dimensions, hardware acceleration, and latency settings. Compare browser decoding with the native decoder result and document the conditions that explain the roughly three-second delay; the issue is resolved when the browser-side cause or required configuration change is established.