[macOS Screen Capture] Incorrect luminance when capturing Rec. 709 (distorted grays)

Operating System Info

Other

Other OS

macOS 14

OBS Studio Version

30.0.0-beta2

OBS Studio Version (Other)

No response

OBS Studio Log URL

https://obsproject.com/logs/UEOzuiEtnMgSnVnw

OBS Studio Crash Log URL

No response

Expected Behavior

Grays ought to be captured with near complete fidelity.

Current Behavior

Grays are markedly distorted.

Steps to Reproduce

1. Add a macOS Screen Capture source
2. Set color space to Rec. 709
3. Create recording
4. Compare color primaries in result to source

Anything else we should know?

I am not 100% sure this is a bug with OBS, but I believe it is a strong possibility and worth triaging. Here are some samples that I invite scrutiny on.

1) is a still from the attached QuickTime Test Pattern. 2) is a still from a macOS Screen Capture window capture recorded in OBS of the window for the test pattern video in QuickTime, recorded in the Rec. 709 output color space. 3) is the same still from an OBS capture, but the OBS capture was recorded using the sRGB output color space. 4) is a still from my own program, which takes in SCK frames in the P3 color space and sends them directly to VideoToolbox, while telling VT that the output color space is Rec. 709 such that VT performs its own mapping.

The thing that I find puzzling here is that OBS sRGB seems to have correct luminance values relative to the reference, but OBS Rec. 709 does not. I also do not believe that this is a ScreenCaptureKit issue, since, with my own program, I can correctly capture the luminance just by taking frames with P3 primaries from ScreenCaptureKit and providing them directly to VideoToolbox (while specifying to VT that they should be mapped to Rec. 709).

Here are the actual recorded samples.

QuickTime Test Pattern:

https://github.com/obsproject/obs-studio/assets/6864788/e3c31489-288d-407a-91c7-8da506575603

OBS sRGB:

https://github.com/obsproject/obs-studio/assets/6864788/49c9661b-3fa1-4ee8-b33a-57da84682a51

OBS Rec. 709:

https://github.com/obsproject/obs-studio/assets/6864788/0083ac6c-6998-4319-bf01-d203ab72aafe

Record.app Rec. 709:

https://github.com/obsproject/obs-studio/assets/6864788/1d5231f8-c3a5-437b-8924-72d2d6987fcf

As a final reference, here is OBS capturing the test pattern in the 709 color space *as a media source* rather than as a window capture. Here, the luminance is captured correctly, which is why I've isolated this issue to macOS Screen Capture.

https://github.com/obsproject/obs-studio/assets/6864788/549f4b37-e15a-4944-8a93-746c6c9414dd

Reproduce with a macOS Screen Capture source set to Rec. 709, record the QuickTime test pattern, and compare its grays with the source and the sRGB capture. Investigate the macOS Screen Capture color conversion path; done means Rec. 709 captures preserve luminance while the existing sRGB and media-source comparisons remain understood.