[BUG]: In current architecture, copying textures to the encoder takes a really long time. It should not.

Package version

2.4.0-exp.8

Environment

* OS: macOS, Windows 11
* Unity version: Unity 2021.3.6f1

Steps To Reproduce

1. On macOS, stream with encoderImplementation=VideoToolbox. This is to demonstrate that the underlying issue is not due to idiosyncrasies of implementing a VideoEncoder interface (as you did with NvEncoder). The accelerated video encoder on macOS comes from a libwebrtc authored factory.
2. Does OnFrameCaptured really need to occur on the Render Thread? Why is a copy and NV12 conversion taking so long?
3. Observe on VideoToolBox the actual encoder step is 1.29ms. Very short.

Is the actual encoding blocking the main thread? It is honestly hard to tell. Based on the profiler data, the last thing to happen before the Main Thread is unblocked is the UnityVideoEncoder.Encode step. This makes sense from the point of view of the code too.

As far as I understand, prior to 2.4.0-exp7 the NvEnc based encoder asked the graphics device to copy the frame into a buffer on the Render Thread, but not encode the frame.

I believe you have dealt with this copying issue a lot. In my experience I had the best latency in the 2.3... series, but I believe that shared the DirectX 12 texture backing the RenderTexture directly with NvEnc, which produced numerous artifacts as the engine wrote to the texture at the same time as the encoder thread read it. I do not 100% understand the underlying issues there.

I think it's essential to deal with this problem. Right now com.unity.webrtc is performing worse than NvFBC (i.e. NVIDIA Game Streaming) in part because of this architectural choice. It affects all architectures.

Profiler data:
macOS Metal with custom rendering profiler data.data.zip

Current Behavior

Poor multithreading of encoding.

Expected Behavior

Encoding that is competitive with NvFBC's performance.

Anything else?

No response