Active output (e.g. virtual cam or NDI) combined with NVENC recording causes high frame render times

Operating System Info

Windows 10

Other OS

No response

OBS Studio Version

30.0.0

OBS Studio Version (Other)

No response

OBS Studio Log URL

https://obsproject.com/logs/b8YJTsBlxy294kBh

OBS Studio Crash Log URL

No response

Expected Behavior

Having both an output active (e.g. virtual cam or NDI) and an active NVENC session (recording or stream) shouldn't increase average frame render times to values a lot higher than what the sum of each feature separately adds to this time. For an empty scene something like 0.4 ms would be reasonable, when each feature alone pushes it from 0.1 ms to 0.2 or 0.3 ms. That's also what I get in OBS 29.1.3.

Current Behavior

Average frame render time goes on a wild rollercoaster ride, to 1.0 ms, up to 6.3, back down to 2 ms, up to 5 ms and so on:

Steps to Reproduce

(just cancel or confirm any wizards or confirmation dialogs that are coming up, it's not explicitly mentioned in the steps below)

1. Start a fresh install of OBS 30.0.0+ with default settings and no scenes or sources on an NVENC capable system
2. Click Start Recording
3. Click Start Virtual Camera
4. Click View -> Stats

"Average time to render frame" will be jumping up and down a lot, including up to crazy values, especially for an empty scene at 720p30:

Only an active output by itself is fine (low render time), only an NVENC recording by itself is fine, and also an active output combined with x264 recording is fine. All of these have consistently low render times (<= 0.3 ms) as would be expected for an empty scene. Really only the combo of output + NVENC is affected, so the above repro works because NVENC is enabled by default. Also tested advanced mode and NVENC with different settings, also H.264, HEVC and AV1, it always occurs regardless of that as long as NVENC is used in any form.

If it is reproduced in an OBS instance with many active sources the problem gets even worse, I've seen 16 ms render time and higher in my production installation.

Anything else we should know?

• with OBS 30 the issue occurs for both Virtual Cam and Teleport or other outputs
• with OBS 29 the issue does not occur for Virtual Cam
• with OBS 29 the issue does occur for Teleport and other outputs, but it takes a while after starting the output and the recording (that's the new part), whereas for OBS 30 it happens immediately
• activating more outputs simultaneously doesn't further increase the frame rendering times
• this is not just a statistics/display issue, with more complex scene and source configurations it actually causes other issues because of high frame render time, e.g. dropped frames
• could reproduce on an RTX 3090 and RTX 4080 with NVIDIA drivers 546.17 and 546.33, as well as a GTX 1080 with driver 546.01

Reproduce the issue from the listed entry points: Start Recording with NVENC, Start Virtual Camera, then open View -> Stats and watch average frame render time. Compare OBS 30.0.0 with OBS 29.1.3 and review the linked OBS log; done means combined output and NVENC no longer cause unstable or excessive render times without regressing the separate-output cases.