AcademySoftwareFoundation / AcademySoftwareFoundation/OpenImageIO
[BUG] thread_pool shutdown dead lock
- Dominant language
- C++
- Stars
- 2.4k
- Forks
- 698
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 48
Description
**Describe the bug**
Occasionally idiff.exe refuses to shutdown on the blender CI environment, previously openexr was to blame here but with recent developments there that specific problem appears to be solved, however a new one has sprung up in its place. I have a process dump from the CI environment. which shows the following stack with only the main thread still being alive.
```
ntdll.dll!NtYieldExecution()
KERNELBASE.dll!SwitchToThread()
[Inline Frame] OpenImageIO_Util.dll!std::this_thread::yield() Line 179
[Inline Frame] OpenImageIO_Util.dll!OpenImageIO_v2_4::atomic_backoff::operator()() Line 146
OpenImageIO_Util.dll!OpenImageIO_v2_4::spin_mutex::lock() Line 219
[Inline Frame] OpenImageIO_Util.dll!std::unique_lock::{ctor}(OpenImageIO_v2_4::spin_mutex &) Line 137
[Inline Frame] OpenImageIO_Util.dll!OpenImageIO_v2_4::pvt::ThreadsafeQueue *>::pop(std::function * &) Line 63
OpenImageIO_Util.dll!OpenImageIO_v2_4::thread_pool::Impl::clear_queue() Line 178
OpenImageIO_Util.dll!OpenImageIO_v2_4::thread_pool::Impl::stop(bool isWait) Line 249
[Inline Frame] OpenImageIO_Util.dll!OpenImageIO_v2_4::thread_pool::Impl::{dtor}() Line 122
OpenImageIO_Util.dll!OpenImageIO_v2_4::thread_pool::Impl::`scalar deleting destructor'(unsigned int)
OpenImageIO_Util.dll!`OpenImageIO_v2_4::default_thread_pool'::`2'::`dynamic atexit destructor for 'shared_pool''()
ucrtbase.dll!(void)()
ucrtbase.dll!__crt_seh_guarded_call::operator()<,(void) &,>()
ucrtbase.dll!_execute_onexit_table()
OpenImageIO_Util.dll!dllmain_crt_process_detach(const bool is_terminating) Line 182
OpenImageIO_Util.dll!dllmain_dispatch(HINSTANCE__ * const instance, const unsigned long reason, void * const reserved) Line 293
ntdll.dll!LdrpCallInitRoutine()
ntdll.dll!LdrShutdownProcess()
ntdll.dll!RtlExitUserProcess()
kernel32.dll!ExitProcessImplementation()
ucrtbase.dll!exit_or_terminate_process()
ucrtbase.dll!common_exit()
idiff.exe!__scrt_common_main_seh() Line 295
kernel32.dll!BaseThreadInitThunk()
ntdll.dll!RtlUserThreadStart()
````
**To Reproduce**
- Almost the same setup as openEXR recently you need busybee from [that repro](https://github.com/OpenImageIO/oiio/pull/3805#issuecomment-1510493759)
- Grab [jpeg2000-rgba-12-jp2-90__from__rgba08.jp2](https://svn.blender.org/svnroot/bf-blender/trunk/lib/tests/imbuf_io/reference/jpeg2000-rgba-12-jp2-90__from__rgba08.jp2) from the blender test suite
- make a little batch file called loop.cmd
```@echo off
:loop_start
echo %TIME%
..\idiff -fail 1 -failpercent 100 -abs -scale 16 -o aaa.diff.png jpeg2000-rgba-12-jp2-90__from__rgba08.jp2 jpeg2000-rgba-12-jp2-90__from__rgba08.jp2
goto loop_start
```
- start `busybee 4` (See #3805 for finding the optimal number of threads to use) in one console
- start `loop.cmd` on another console
- Wait anywhere between 30 seconds and 30 minutes (yeahh... sorry about that...this _is_ the best repro i have sadly) until it gets stuck.
**Platform information:**
- OIIO branch/version: v2.4.11.0
- OS: Windows 10
- C++ compiler: MSVC 2019
- Any non-default build flags when you build OIIO: Too many flags to list, build config can be found over here in the [blender repository](https://projects.blender.org/blender/blender/src/branch/main/build_files/build_environment/cmake/openimageio.cmake#L44)
Contributor guide
Research direction
Reproduce the shutdown hang on Windows 10 using idiff, busybee 4, loop.cmd, and the linked JPEG 2000 file. Start by inspecting the thread_pool::Impl stop/clear_queue path and the ThreadsafeQueue::pop stack shown in the dump; done means the repeated idiff process exits without deadlocking during shutdown.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100