AcademySoftwareFoundation / AcademySoftwareFoundation/OpenImageIO

[BUG] thread_pool shutdown dead lock

Open
#3,851 8 comments 0 reactions 0 assignees View on GitHub
bug internals
Dominant language
C++
Stars
2.4k
Forks
698
Avg merge
3d 9h
Merged PRs (30d)
48

Description

**Describe the bug**

Occasionally idiff.exe refuses to shutdown on the blender CI environment, previously openexr was to blame here but with recent developments there that specific problem appears to be solved, however a new one has sprung up in its place. I have a process dump from the CI environment. which shows the following stack with only the main thread still being alive.

```
ntdll.dll!NtYieldExecution()
KERNELBASE.dll!SwitchToThread()
[Inline Frame] OpenImageIO_Util.dll!std::this_thread::yield() Line 179
[Inline Frame] OpenImageIO_Util.dll!OpenImageIO_v2_4::atomic_backoff::operator()() Line 146
OpenImageIO_Util.dll!OpenImageIO_v2_4::spin_mutex::lock() Line 219
[Inline Frame] OpenImageIO_Util.dll!std::unique_lock::{ctor}(OpenImageIO_v2_4::spin_mutex &) Line 137
[Inline Frame] OpenImageIO_Util.dll!OpenImageIO_v2_4::pvt::ThreadsafeQueue *>::pop(std::function * &) Line 63
OpenImageIO_Util.dll!OpenImageIO_v2_4::thread_pool::Impl::clear_queue() Line 178
OpenImageIO_Util.dll!OpenImageIO_v2_4::thread_pool::Impl::stop(bool isWait) Line 249
[Inline Frame] OpenImageIO_Util.dll!OpenImageIO_v2_4::thread_pool::Impl::{dtor}() Line 122
OpenImageIO_Util.dll!OpenImageIO_v2_4::thread_pool::Impl::`scalar deleting destructor'(unsigned int)
OpenImageIO_Util.dll!`OpenImageIO_v2_4::default_thread_pool'::`2'::`dynamic atexit destructor for 'shared_pool''()
ucrtbase.dll!(void)()
ucrtbase.dll!__crt_seh_guarded_call::operator()<,(void) &,>()
ucrtbase.dll!_execute_onexit_table()
OpenImageIO_Util.dll!dllmain_crt_process_detach(const bool is_terminating) Line 182
OpenImageIO_Util.dll!dllmain_dispatch(HINSTANCE__ * const instance, const unsigned long reason, void * const reserved) Line 293
ntdll.dll!LdrpCallInitRoutine()
ntdll.dll!LdrShutdownProcess()
ntdll.dll!RtlExitUserProcess()
kernel32.dll!ExitProcessImplementation()
ucrtbase.dll!exit_or_terminate_process()
ucrtbase.dll!common_exit()
idiff.exe!__scrt_common_main_seh() Line 295
kernel32.dll!BaseThreadInitThunk()
ntdll.dll!RtlUserThreadStart()
````

**To Reproduce**

- Almost the same setup as openEXR recently you need busybee from [that repro](https://github.com/OpenImageIO/oiio/pull/3805#issuecomment-1510493759)
- Grab [jpeg2000-rgba-12-jp2-90__from__rgba08.jp2](https://svn.blender.org/svnroot/bf-blender/trunk/lib/tests/imbuf_io/reference/jpeg2000-rgba-12-jp2-90__from__rgba08.jp2) from the blender test suite
- make a little batch file called loop.cmd
```@echo off
:loop_start
echo %TIME%
..\idiff -fail 1 -failpercent 100 -abs -scale 16 -o aaa.diff.png jpeg2000-rgba-12-jp2-90__from__rgba08.jp2 jpeg2000-rgba-12-jp2-90__from__rgba08.jp2
goto loop_start
```
- start `busybee 4` (See #3805 for finding the optimal number of threads to use) in one console
- start `loop.cmd` on another console
- Wait anywhere between 30 seconds and 30 minutes (yeahh... sorry about that...this _is_ the best repro i have sadly) until it gets stuck.

**Platform information:**
- OIIO branch/version: v2.4.11.0
- OS: Windows 10
- C++ compiler: MSVC 2019
- Any non-default build flags when you build OIIO: Too many flags to list, build config can be found over here in the [blender repository](https://projects.blender.org/blender/blender/src/branch/main/build_files/build_environment/cmake/openimageio.cmake#L44)

Contributor guide

Open the contributing guide

Research direction

Reproduce the shutdown hang on Windows 10 using idiff, busybee 4, loop.cmd, and the linked JPEG 2000 file. Start by inspecting the thread_pool::Impl stop/clear_queue path and the ThreadsafeQueue::pop stack shown in the dump; done means the repeated idiff process exits without deadlocking during shutdown.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.