Deadlock between threads using data->sources_mutex and current_lua_script->mutex

Operating System Info

Windows 10

Other OS

No response

OBS Studio Version

27.0.1

OBS Studio Version (Other)

Also occurs with home-build using current source

OBS Studio Log URL

OBS hangs; log file doesn't show this

OBS Studio Crash Log URL

No response

Expected Behavior

Non-looped Lua scripts shouldn't cause a hang or deadlock

Current Behavior

Deadlock in at least the cases described below

Steps to Reproduce

I am triggering a mutex deadlock, by running a diagnostic Lua script that hooks to frontend events and source activate/deactivate and show/hide signals. The script logs changes in scenes and sources, and is sometimes useful to help me debug other scripts. (Script attached)

logorrhea.lua.txt

After a variable number of transitions (1 to over 50, and sometimes before the UI is even shown) the UI freezes. I still see "thread XXXX has exited" etc., so other threads are still running.

This happens both with current source and in version 27.0.1. Seems to occur more frequently in a debug build.

In a debug build of current (as of 14 Sept 2021) source, I break OBS after the hang giving these results:

Windows thread ID 3DD4 (Main Thread) is pthread_t* 0x000001d3db5b5f70
Waiting for mutex 0x000001d3dd76a4d0 owned by 0x000001d3dfa25600 (This is data->sources_mutex)
Stack has

w32-pthreads.dll!pthread_mutex_lock(pthread_mutex_t_ * * mutex) Line 127 C
obs.dll!obs_enum_sources(bool(*)(void *, obs_source *) enum_proc, void * param) Line 1451 C
obs-scripting.dll!enum_sources(lua_State * script) Line 589 C
[External Code]
obs-scripting.dll!call_func_(lua_State * script, int reg_idx, int args, int rets, const char * func, const char * display_name) Line 234 C
obs-scripting.dll!frontend_event_callback(obs_frontend_event event, void * priv) Line 203 C
obs64.exe!OBSStudioAPI::on_event(obs_frontend_event event) Line 655 C++
obs64.exe!OBSBasic::TransitionStopped() Line 328 C++
obs64.exe!OBSBasic::qt_static_metacall(QObject * _o, QMetaObject::Call _c, int id, void * * a) Line 1197 C++
[External Code]
obs64.exe!run_program(std::basic_fstream<char,std::char_traits> & logFile, int argc, char * * argv) Line 2131 C++
obs64.exe!main(int argc, char * * argv) Line 2819 C++
obs64.exe!WinMain(HINSTANCE * formal, HINSTANCE * __formal, char * __formal, int __formal) Line 97 C++

Windows thread ID 3074 (libobs: graphics thread) is pthread_t* 0x000001d3dfa25600
Waiting for mutex 0x000001d3e49131d0 owned by 0x000001d3db5b5f70 (This is current_lua_script->mutex)
Stack has

w32-pthreads.dll!pthread_mutex_lock(pthread_mutex_t_ * * mutex) Line 127 C
obs-scripting.dll!calldata_signal_callback(void * priv, calldata * cd) Line 433 C
obs.dll!signal_handler_signal(signal_handler * handler, const char * signal, calldata * params) Line 318 C
obs.dll!obs_source_dosignal(obs_source * source, const char * signal_obs, const char * signal_source) Line 820 C
obs.dll!hide_source(obs_source * source) Line 1063 C
obs.dll!obs_source_video_tick(obs_source * source, float seconds) Line 1191 C
obs.dll!tick_sources(unsigned __int64 cur_time, unsigned __int64 last_time) Line 71 C
obs.dll!obs_graphics_thread_loop(obs_graphics_context * context) Line 962 C
obs.dll!obs_graphics_thread(void * param) Line 1052 C
w32-pthreads.dll!ptw32_threadStart(void * vthreadParms) Line 225 C

So classic deadlock.

Another run hangs during startup, before UI is visible:
Windows thread ID 164C (Main Thread) is pthread_t* 0x000001ddbb5ce080
Waiting for mutex 0x000001ddbd630a00 owned by 0x000001ddbfaeba30

w32-pthreads.dll!pthread_mutex_lock(pthread_mutex_t_ * * mutex) Line 127 C
obs.dll!obs_enum_sources(bool(*)(void *, obs_source *) enum_proc, void * param) Line 1451 C
obs-scripting.dll!enum_sources(lua_State * script) Line 589 C
[External Code]
obs-scripting.dll!call_func_(lua_State * script, int reg_idx, int args, int rets, const char * func, const char * display_name) Line 234 C
obs-scripting.dll!frontend_save_callback(obs_data * save_data, bool saving, void * priv) Line 252 C
obs64.exe!OBSStudioAPI::on_load(obs_data * settings) Line 626 C++
obs64.exe!OBSBasic::LoadData(obs_data * data, const char * file) Line 1154 C++
obs64.exe!OBSBasic::Load(const char * file) Line 968 C++
obs64.exe!OBSBasic::OBSInit() Line 1880 C++
obs64.exe!OBSApp::OBSInit() Line 1462 C++
obs64.exe!run_program(std::basic_fstream<char,std::char_traits> & logFile, int argc, char * * argv) Line 2126 C++
obs64.exe!main(int argc, char * * argv) Line 2819 C++
obs64.exe!WinMain(HINSTANCE__ * formal, HINSTANCE * __formal, char * __formal, int __formal) Line 97 C++

Windows thread ID 1A08 is pthread_t* 0x000001ddbfaeba30
Waiting for mutex 0x000001ddc4800800 owned by 0x000001ddbb5ce080

w32-pthreads.dll!pthread_mutex_lock(pthread_mutex_t_ * * mutex) Line 127 C
obs-scripting.dll!calldata_signal_callback(void * priv, calldata * cd) Line 433 C
obs.dll!signal_handler_signal(signal_handler * handler, const char * signal, calldata * params) Line 318 C
obs.dll!obs_source_dosignal(obs_source * source, const char * signal_obs, const char * signal_source) Line 820 C
obs.dll!activate_source(obs_source * source) Line 1042 C
obs.dll!obs_source_video_tick(obs_source * source, float seconds) Line 1211 C
obs.dll!tick_sources(unsigned __int64 cur_time, unsigned __int64 last_time) Line 71 C
obs.dll!obs_graphics_thread_loop(obs_graphics_context * context) Line 962 C
obs.dll!obs_graphics_thread(void * param) Line 1052 C
w32-pthreads.dll!ptw32_threadStart(void * vthreadParms) Line 225 C

Function tick_sources in obs-video.c calls obs_source_video_tick for all sources while holding data->sources_mutex

Anything else we should know?

Seems similar to 2018 Windows Support forum post:
https://obsproject.com/forum/threads/obs-sometimes-freezes-when-reloading-script-with-lua-filter.84423/

Start with obs-video.c, especially tick_sources, and the obs-scripting paths enum_sources, call_func_, and calldata_signal_callback shown in the deadlock stacks. Reproduce on Windows with the attached logorrhea.lua script, preferably in a debug build, and inspect both mutex wait paths. Done means non-looped Lua scripts no longer freeze OBS during source transitions or startup.