Incorrect capture area size when capturing a system DPI aware process with a different system DPI than the OBS process

Operating System Info

Windows 11

Other OS

No response

OBS Studio Version

29.0.2

OBS Studio Version (Other)

No response

OBS Studio Log URL

None

OBS Studio Crash Log URL

No response

Expected Behavior

OBS can correctly get the capture area size of all windows in any DPI awareness context.

Current Behavior

If you use OBS to capture a system DPI aware window via BitBlt, and the target process has a different system DPI value than the OBS process, this line

https://github.com/obsproject/obs-studio/blob/e9ef38e3d38e08bffcabbb59230b94baa41ede96/plugins/win-capture/window-capture.c#L610-L611

will fail with ERROR_INVALID_PARAMETER and the return value will be NULL.

So the following GetClientRect() will still use the current DPI awareness context to calculate the coordinates, leading to an incorrect capture area size.

Steps to Reproduce

1. Launch a system DPI aware program whose windows can be captured with BitBlt.
2. Change the DPI value of the primary monitor (by changing the display settings or by switching to another monitor).
3. Launch OBS and capture the window with window source via BitBlt.
4. The captured area for the window will either be smaller or larger than the actual client area.

Anything else we should know?

When using BitBlt to capture a window, OBS tries to apply the DPI awareness context of the target window to its own thread, so that it can get the correct coordinates for the target window DC.

https://github.com/obsproject/obs-studio/blob/e9ef38e3d38e08bffcabbb59230b94baa41ede96/plugins/win-capture/window-capture.c#L604-L612

This works most of the time, when the target process is DPI unaware or is per-monitor DPI aware.

However, if the target process is system DPI aware, and the target process has a different system DPI value than OBS, Windows will refuse to apply the system-aware context with a different system DPI, possibly because system DPI value is a per-process setting (see GetSystemDpiForProcess). You can still get the DPI value the target window is using, however, with GetDpiForWindow.

(Each process can have different system DPI values if you allow Windows to "fix scaling for apps" automatically. If that is enabled, the system DPI value of a process will be the DPI of the primary monitor when the process started, rather than when the user logged in, so that users can restart system DPI aware programs to fix their blurriness after changing the primary monitor DPI)

Here's some other possible ways to calculate the correct coordinates.

• Get the DPI of the target window using GetDpiForWindow, get the DPI of the monitor the target window is on using GetDpiForMonitor, then calculate values using code like this: x = MulDiv(x, targetDpi, monitorDpi);
• Use functions like PhysicalToLogicalPointForPerMonitorDPI. According to the docs, this function works on Windows 8.1. However, this function only works if the provided point (in screen coordinates) is inside the window rectangle. Otherwise, it will refuse to convert the coordinates and return FALSE. So in order to use it, you will have to first convert the client rectangle to screen coordinates.

However, both methods can sometimes result in sizes off by 1 pixel compared to what the target process gets using GetClientRect.

WGC is generally better and doesn't have this issue, but for windows that support BitBlt, BitBlt actually copies the bitmap that hasn't been stretched by DWM, so it can capture crisp images of DPI unaware windows though they are still blurry on screen.

Start in plugins/win-capture/window-capture.c around lines 604-612, where OBS applies the target window's DPI awareness context and calls GetClientRect(). Reproduce the issue with a system DPI aware target whose DPI differs from OBS, then inspect GetDpiForWindow and the coordinate-conversion alternatives mentioned in the report. Done means BitBlt captures the correct client area size across differing system DPI values.