gs_clear crash when calling gs_texrender_begin + obs_source_video_render in Qt + libobs integration

Operating System Info

Windows 11

Other OS

No response

OBS Studio Version

Git

OBS Studio Version (Other)

msvc build

OBS Studio Log URL

无

OBS Studio Crash Log URL

No response

Expected Behavior

I'm integrating libobs (OBS Studio 30.x) into a Qt 6.6 application to capture rendered scene frames and encode them. However, when I call gs_clear(GS_CLEAR_COLOR, nullptr, 0.0f, 0.0f) during capture, the application crashes. This happens even after calling gs_texrender_begin() and obs_source_video_render(). Here's the high-level capture code:

Current Behavior

Hi,
First of all, thank you for maintaining such a powerful open-source project. I'm currently working on integrating libobs into a Qt-based application for real-time scene rendering and encoding. However, I’ve encountered a crash when using gs_clear() after gs_texrender_begin(), and I would sincerely appreciate any guidance or suggestions you can offer.

Steps to Reproduce

1. Git clone the master branch and build it using MSVC on Windows x64
2. Use Qt to create a QML project and integrate the libobs library via CMake
   3.Start a worker thread and initialize OBS with obs_startup("en-US", nullptr, nullptr)
   4.Create an obs_scene_t and add an obs_source_t to it
   5.Create a gs_texrender_t and execute gs_clear(GS_CLEAR_COLOR, nullptr, 0.0f, 0.0f);

Anything else we should know?

void ScreenCaptureOBS::run()
{   if (!ObsManager::instance().initialize()) {
        qWarning() << "OBS failed to initialize.";
        return ;
    }
    if (!init()) {
        qWarning() << "Failed to init ScreenCaptureOBS.";
        return ;
    }
    while (_running) {
        refreshWindowSources();
        captureFrame();
        msleep(_deviceInfoQObject->frameRate());
    }
}
void ScreenCaptureOBS::refreshWindowSources()
{
    QMap<HWND, bool> existing;
    for (auto hwnd : _windowSources.keys()) {
        existing[hwnd] = false;
    }

    EnumWindows([](HWND hwnd, LPARAM lParam) -> BOOL {
        auto* self = reinterpret_cast<ScreenCaptureOBS*>(lParam);

        DWORD pid = 0;
        GetWindowThreadProcessId(hwnd, &pid);
        if (pid == self->_processId && IsWindowVisible(hwnd)) {
            if (!self->_windowSources.contains(hwnd)) {
                wchar_t title[512] = {0};
                GetWindowTextW(hwnd, title, sizeof(title) / sizeof(wchar_t));
                qDebug() << "HWND:" << hwnd << "Title:" << QString::fromWCharArray(title);
                obs_enter_graphics();
                obs_data_t* settings = obs_data_create();
                obs_data_set_string(settings, "window", QString::asprintf("%p", hwnd).toUtf8().data());
                obs_source_t* source = obs_source_create("window_capture", "WindowCapture", settings, nullptr);
                if (source) {
                    // QMutexLocker locker(&self->_sceneMutex);
                    auto* item = obs_scene_add(self->_scene, source);
                    obs_source_t* sceneSource = obs_scene_get_source(self->_scene);
                    obs_sceneitem_set_visible(item, true);
                    self->_windowSources.insert(hwnd, source);
                }
                obs_data_release(settings);
                obs_leave_graphics();
            }
            self->_windowSources[hwnd]; // mark existing
        }
        return TRUE;
    }, reinterpret_cast<LPARAM>(this));
    // Remove dead windows
    for (auto it = _windowSources.begin(); it != _windowSources.end();) {
        if (!IsWindow(it.key()) || !IsWindowVisible(it.key())) {
            // Remove source from the scene before releasing it
            obs_enter_graphics();
            obs_source_t* source = it.value();

            if (source) {
                // Find the scene item associated with the source
                obs_sceneitem_t* item = obs_scene_find_source(this->_scene, obs_source_get_name(source));
                if (item) {
                    obs_sceneitem_remove(item); // Remove item from scene
                }
                obs_source_release(source); // Release the source
            }
            obs_leave_graphics();
            it = _windowSources.erase(it); // Remove it from the list
        } else {
            ++it;
        }
    }
    obs_enter_graphics();
    obs_source_t* sceneSource = obs_scene_get_source(_scene);
    obs_leave_graphics();
}
void ScreenCaptureOBS::captureFrame()
{
    if (!_scene) {
        qWarning() << "Scene is null.";
        return;
    }

    obs_source_t* sceneSource = obs_scene_get_source(_scene);
    if (!sceneSource) {
        qWarning() << "Scene source is null.";
        return;
    }

    uint32_t sceneWidth = obs_source_get_width(sceneSource);
    uint32_t sceneHeight = obs_source_get_height(sceneSource);

    if (sceneWidth == 0 || sceneHeight == 0) {
        qWarning() << "Scene size is invalid.";
        return;
    }
    if (!_texrender) {
        obs_enter_graphics();
        _texrender = gs_texrender_create(GS_RGBA, GS_ZS_NONE);
        obs_leave_graphics();

        if (!_texrender) {
            qWarning() << "Failed to create texrender.";
            return;
        }
    }

    obs_enter_graphics();
    bool success = gs_texrender_begin(_texrender, sceneWidth, sceneHeight);
    if (!success) {
        obs_leave_graphics();
        qWarning() << "Failed to begin texrender.";
        return;
    }

    gs_clear(GS_CLEAR_COLOR, nullptr, 0.0f, 0.0f);
    obs_source_video_render(sceneSource);
    gs_texrender_end(_texrender);

    gs_texture_t* tex = gs_texrender_get_texture(_texrender);
    if (!tex) {
        obs_leave_graphics();
        qWarning() << "Failed to get texture from texrender.";
        return;
    }

    uint8_t* data = nullptr;
    uint32_t linesize = 0;
    if (!gs_texture_map(tex, &data, &linesize)) {
        obs_leave_graphics();
        qWarning() << "Failed to map texture.";
        return;
    }

    QImage frame(data, sceneWidth, sceneHeight, linesize, QImage::Format_ARGB32_Premultiplied);

    {
        QMutexLocker locker(&_frameMutex);
        _frame = frame.copy();
    }

    obs_leave_graphics();

    initEncoder(sceneWidth, sceneHeight, _deviceInfoQObject->frameRate(), _deviceInfoQObject->frameRate());
    encodeAndPushFrame(_frame);

    obs_enter_graphics();
    gs_texture_unmap(tex);
    obs_leave_graphics();
}

Start with captureFrame() and the sequence around obs_enter_graphics(), gs_texrender_begin(), gs_clear(), obs_source_video_render(), and gs_texrender_end(). Review how the Qt worker thread initializes libobs and runs the capture loop, then reproduce the crash using the listed Windows/MSVC and Qt integration steps. Done means the reported capture sequence no longer crashes and frames can be obtained from the texture render.