[BUG]: Video frames interleaved/mixed between multiple video tracks when using NVENC hardware encoding

Package version

Others

Environment

* Package version: 3.0.0
* OS: Windows 10 (x64)
* Unity version: 2022.3.20f1
* GPU: RTX 3050
* Graphics API: D3D11
* Platform: Editor / Windows Standalone (sender side)

Steps To Reproduce

1. Create one RTCPeerConnection and add multiple VideoStreamTracks that share the same resolution and pixel format (in our case: an IR stream, an RGB stream and a depth stream, all e.g. 640x480/1280x960, H264).
2. Configure all tracks to use the NVIDIA hardware encoder (NVENC, H264).
3. Stream the tracks to a WebRTC client (GStreamer webrtcbin or the Unity MultiVideoStreamReceiver) and observe the IR/RGB video.
4. Reproducibility is intermittent; the wrong frames appear in bursts. Interleaving happens on the sender side.

Current Behavior

The client displays the IR and RGB streams interleaved/periodically swapped: a few IR frames are shown in the RGB video and vice versa. The encoder input on the sender side is verified correct (frames captured from the source textures before encoding are clean and never mixed), which isolates the corruption to the encoding stage.

The corruption is confirmed to be in the encoded bitstream itself, not in the receiver demuxing:

• We attached an RTCRtpScriptTransform (insertable streams) to the IR sender's RTCVideoSender and dumped the encoded H264 payloads by SSRC to local files.
• The saved .h264 files for the IR track contain interleaved RGB frames (both files show the same symptom, reproduced with different SSRCs).
• The same encoded stream is decoded wrong on any receiver type, so the receiver is not at fault.

Expected Behavior

Each VideoStreamTrack must be encoded independently; frames from one track must never appear inside another track's encoded output.

Anything else?

Root cause analysis (native plugin)

The interleaving can be fully explained by a buffer-reuse race in the shared GpuMemoryBufferPool, combined with unsynchronized asynchronous GPU copies:

1. Pool reuse does not bind a buffer to its source texture.

   GpuMemoryBufferPool::GetOrCreateFrameResources() reuses a pooled FrameResources when !resources->IsUsed() && AreFrameResourcesCompatible(resources, size, format), and AreFrameResourcesCompatible() compares only GetSize() and GetFormat() — the source texture pointer ptr (the argument of CreateFrame/GetOrCreateFrameResources) is never stored or compared. Therefore a buffer freed by track A (IR) can immediately be rebound to track B (RGB) in the same GPU frame cycle.

   rtc::scoped_refptr<GpuMemoryBufferInterface> GpuMemoryBufferPool::GetOrCreateFrameResources(
       NativeTexPtr ptr, const Size& size, UnityRenderingExtTextureFormat format)
   {
       std::lock_guard<std::mutex> lock(mutex_);
       for (auto it = resourcesPool_.begin(); it != resourcesPool_.end(); ++it)
       {
           FrameResources* resources = it->get();
           if (!resources->IsUsed() && AreFrameResourcesCompatible(resources, size, format))
           {
               GpuMemoryBufferFromUnity* buffer = static_cast<GpuMemoryBufferFromUnity*>(resources->buffer_.get());
               if (!buffer->ResetSync()) { continue; }
               if (!buffer->CopyBuffer(ptr)) { continue; }   // overwrites the buffer with the OTHER track
               resources->MarkUsed(clock_->CurrentTime());
               return resources->buffer_;
           }
       }
       ...
   }
   
   bool GpuMemoryBufferPool::AreFrameResourcesCompatible(
       const FrameResources* resources, const Size& size, UnityRenderingExtTextureFormat format)
   {
       return resources->buffer_->GetSize() == size && resources->buffer_->GetFormat() == format;
   }
   

   References: Plugin~/WebRTCPlugin/GpuMemoryBufferPool.cpp

2. WaitSync waits for the wrong signal when a buffer is reused across tracks.

   In D3D11GraphicsDevice, each copy does UpdateSyncCount() then Signal(fence, syncCount). WaitSync(texture) reads texture->GetSyncCount() at wait time. When track A's encoder thread is about to encode and track B has already rebound the shared buffer and signalled a new value, track A's WaitSync waits for track B's signal, i.e. it synchronizes against the other track's copy. The intended "this buffer is ready with my frame" guarantee is lost.

   bool D3D11GraphicsDevice::WaitSync(const ITexture2D* texture)
   {
       const D3D11Texture2D* d3d11Texture = static_cast<const D3D11Texture2D*>(texture);
       const uint64_t value = d3d11Texture->GetSyncCount();   // may already belong to the other track's copy
       ...
   }
   

   References: Plugin~/WebRTCPlugin/GraphicsDevice/D3D11/D3D11GraphicsDevice.cpp (CopyResourceV / CopyResourceFromNativeV / WaitSync / Signal)

3. The NVENC input copy is asynchronous and not awaited.

   NvEncoderImpl::Encode() calls CopyResource() which ends in NvEncoderCudaWithCUarray::CopyToDeviceFrame() → cuMemcpy2DAsync(&m, stream), then submits EncodeFrame() and returns immediately. Nothing guarantees the CUDA copy has read the buffer contents before the VideoFrame is released, which returns the buffer to the pool (OnReturnBuffer → MarkUnused) where the other track overwrites it.

   CUDA_DRVAPI_CALL(stream == NULL ? cuMemcpy2D(&m) : cuMemcpy2DAsync(&m, stream));  // async
   

   References:

   • Plugin~/WebRTCPlugin/Codec/NvCodec/NvEncoderImpl.cpp (Encode, CopyResource)
   • Plugin~/WebRTCPlugin/Codec/NvCodec/NvEncoderCudaWithCUarray.cpp (CopyToDeviceFrame)

Combined, the race is:

Render thread (IR):   reuse pooled buffer B  -> CopyBuffer(IR)   -> Signal(B, N)
Encode thread (IR):   WaitSync(B) -> cuMemcpy2DAsync(B -> CUarray) -> EncodeFrame() -> return
                                                              (copy NOT awaited)
B released back to pool (MarkUnused)
Render thread (RGB):  reuse buffer B (compatible: same size/format) -> CopyBuffer(RGB) overwrites B -> Signal(B, N+1)
GPU executes: IR's pending async copy now reads RGB content  -> IR encoder emits an RGB frame

This also explains why it only reproduces when the tracks have the same resolution and format (only then does the pool reuse a buffer across tracks) and why the encoder input (cachedRT) is always correct — the corruption happens inside the plugin between the input buffer and the encoded output.

Suggested fixes (any one should prevent the interleaving)

1. Bind each pooled buffer to its source texture, e.g. store ptr in FrameResources and require resources->sourcePtr == ptr in AreFrameResourcesCompatible (or keep a dedicated pool per VideoStreamTrack).
2. In NvEncoderImpl::Encode, wait for the CUDA copy to complete (synchronize the CUDA stream with the D3D11 fence, or make CopyToDeviceFrame use a synchronous copy for reused buffers) before the frame/buffer is released back to the pool.
3. In WaitSync, capture/validate the sync value that belongs to the current buffer owner rather than reading the texture's mutable counter.