Deadlock if encounter encoder error

Operating System Info

Windows 11

Other OS

No response

OBS Studio Version

30.0.0

OBS Studio Version (Other)

No response

OBS Studio Log URL

None

OBS Studio Crash Log URL

No response

Expected Behavior

Not deadlock

Current Behavior

Deadlock if encounter encoder error

Steps to Reproduce

1. Start streaming/recording
2. Deadlock if encounter encoder error

Reason

If encounter encoder error, The callstack like below:

 	ntdll.dll!NtWaitForSingleObject()
 	KernelBase.dll!WaitForSingleObjectEx()
	obs.dll!os_event_wait(os_event_data * event)
 	obs.dll!stop_gpu_encode(obs_encoder * encoder)
 	obs.dll!remove_connection(obs_encoder * encoder, bool shutdown)
 	obs.dll!full_stop(obs_encoder * encoder)
 	obs.dll!send_off_encoder_packet(obs_encoder * encoder, bool success, bool received, encoder_packet * pkt)
 	obs.dll!gpu_encode_thread(obs_core_video_mix * video)

Code snipet：

static void *gpu_encode_thread(struct obs_core_video_mix *video)
{
os_event_reset(video->gpu_encode_inactive);
send_off_encoder_packet(encoder, success, received,
			&pkt);
os_event_signal(video->gpu_encode_inactive);
}

void stop_gpu_encode(obs_encoder_t *encoder)
{
os_event_wait(video->gpu_encode_inactive);
}

In stop_gpu_encode, it will wait video->gpu_encode_inactive, but video->gpu_encode_inactive is reset in gpu_encode_thread in same thread.

Not 100% reproduced，like below, full_stop will invoke obs_output_force_stop, then invoke end_data_capture_thread, if all callback is removed from encoder, it will also invoke stop_gpu_encode in end_data_capture_thread.
if invoked stop_gpu_encode in end_data_capture_thread before gpu_encode_thread, then no problem.

void full_stop(struct obs_encoder *encoder)
{
	if (encoder) {
		pthread_mutex_lock(&encoder->outputs_mutex);
		for (size_t i = 0; i < encoder->outputs.num; i++) {
			struct obs_output *output = encoder->outputs.array[i];
			obs_output_force_stop(output);

			pthread_mutex_lock(&output->interleaved_mutex);
			output->info.encoded_packet(output->context.data, NULL);
			pthread_mutex_unlock(&output->interleaved_mutex);
		}
		pthread_mutex_unlock(&encoder->outputs_mutex);

		pthread_mutex_lock(&encoder->callbacks_mutex);
		da_free(encoder->callbacks);
		pthread_mutex_unlock(&encoder->callbacks_mutex);

		remove_connection(encoder, false);
		encoder->initialized = false;
	}
}

I think

		pthread_mutex_lock(&encoder->callbacks_mutex);
		da_free(encoder->callbacks);
		pthread_mutex_unlock(&encoder->callbacks_mutex);

		remove_connection(encoder, false);

Can be remove from full_stop, because other thread will check if all callback is removed from encoder, It will also invoke remove_connection

Anything else we should know?

No response

Start with the gpu_encode_thread, stop_gpu_encode, and full_stop call paths shown in the report, then trace how encoder errors and callback removal interact on Windows. Reproduce an encoder error during streaming or recording and verify that shutdown completes without waiting indefinitely; the fix should avoid the reported deadlock without breaking normal encoder shutdown.