[D3D] Possible conflict between overlays and widgets

Description

There is possible conflict between overlays and widgets in both drivers

• d3d11
• d3d12

Actual behavior

If overlays enabled widgets such as frame rate will not get draw inside the game

Steps to reproduce the bug

1. Enable onscreen notification such as (Frame rate)
2. Start core
3. Enable overlay and select any preset
4. Back to the game
5. Frame rate don't render anymore

Version/Commit

• RetroArch: 1.19.1

Environment information

• OS: Windows
• Driver: DirectX 11 & 12

Details

Widgets usually called right after overlays (if overlay_behind_menu is not enabled):

#ifdef HAVE_OVERLAY
   if ((d3d11->flags & D3D11_ST_FLAG_OVERLAYS_ENABLE) && !overlay_behind_menu)
      d3d11_render_overlay(d3d11);
#endif

#ifdef HAVE_GFX_WIDGETS
   if (widgets_active)
   {
      context->lpVtbl->RSSetViewports(context, 1, &d3d11->viewport);
      gfx_widgets_frame(video_info);
   }
#endif

Overlays render d3d11_render_overlay will set a buffer:

{
   UINT stride = sizeof(d3d11_sprite_t);
   UINT offset = 0;
   d3d11->context->lpVtbl->IASetVertexBuffers(
         d3d11->context, 0, 1, &d3d11->overlays.vbo, &stride, &offset);
}

Now this buffer apparently isn't cleared or set again after that,

widgets gfx_widgets_frame simply use gfx_display_d3d11_draw which don't usually set sprites buffer as I inspected

so the debug will show warning says:

D3D11 WARNING: ID3D11DeviceContext::Draw: Vertex Buffer at the input vertex slot 0 is not big enough for what the Draw*() call expects to traverse..

because buffer was set to d3d11->overlays.vbo before and widgets call is not probably intended to use it.

also there is small problem regarding to overlay_behind_menu:

#ifdef HAVE_OVERLAY
   if ((d3d11->flags & D3D11_ST_FLAG_OVERLAYS_ENABLE) && overlay_behind_menu)
      d3d11_render_overlay(d3d11);
#endif

// ....
// Here menu and osd code
// ...

#ifdef HAVE_OVERLAY
   if ((d3d11->flags & D3D11_ST_FLAG_OVERLAYS_ENABLE) && !overlay_behind_menu)
      d3d11_render_overlay(d3d11);
#endif

Not sure what this option do and in what cases it's used,

so I believe this should be always behind the menu as default (unless it made for something I'm not aware for it)

I thought it will render the overlay while menu visible, but the code logic will skip both calls if menu visible

For the part related to overlay_behind_menu turns out there was option called hide in menu and it was on, didn't notice that.

Possible fixes

In anyway there are few ways to fix it:

Option 1

Reset the buffer at the end of d3d11_render_overlay

// Reset the vertex buffer state to prevent it from affecting other draw calls
ID3D11Buffer* nullBuffer = NULL;
UINT zeroOffset = 0;
d3d11->context->lpVtbl->IASetVertexBuffers(d3d11->context, 0, 1, &nullBuffer, &zeroOffset, &zeroOffset);

Option 2

If widgets require d3d11->sprites.vbo so we can only set it back before calling widgets

if (widgets_active)
{
   context->lpVtbl->RSSetViewports(context, 1, &d3d11->viewport);
   UINT stride = sizeof(d3d11_sprite_t);
   UINT offset = 0;
   context->lpVtbl->IASetVertexBuffers(
      context, 0, 1, &d3d11->sprites.vbo, &stride, &offset);
   gfx_widgets_frame(video_info);
}

Source lines

• render_overlay d3d11.c - d3d12.c
• call location d3d11.c - d3d12.c

Start with render_overlay and its call sites in gfx/drivers/d3d11.c and gfx/drivers/d3d12.c, especially the lines linked in the issue. Reproduce on Windows with overlays and the frame-rate widget enabled, then inspect the Direct3D debug warning and widget draw path. Done means widgets render correctly with overlays enabled in both drivers without the vertex-buffer warning.