Clang and LLVM tests fail under TSan

I ran check-clang and check-llvm under Tsan for the first time in a while, and see that there are several failures in both test suites. I recall running this the last time either last April or maybe in late September. Are these known/tracked issues?

I'm including one of the errors below from clang-scan-deps, which seems disproportionately affected, based on the number of test failures. I do still see errors int he LLVM tests and unit tests under TSan.

The stack trace seems to indicate that there's a race w/ the destructor running in the main thread, which AFAICT shouldn't be possible. Someone that looks at TSan failures more might have a bit more insight.

Error:
```
********************
FAIL: Clang :: ClangScanDeps/strip-codegen-args.m (3714 of 23873)
******************** TEST 'Clang :: ClangScanDeps/strip-codegen-args.m' FAILED ********************
Exit Code: 66

Command Output (stdout):
--
# RUN: at line 1
rm -rf /usr/local/google/home/paulkirth/llvm-fork/build/tools/clang/test/ClangScanDeps/Output/strip-codegen-args.m.tmp
# executed command: rm -rf /usr/local/google/home/paulkirth/llvm-fork/build/tools/clang/test/ClangScanDeps/Output/strip-codegen-args.m.tmp
# RUN: at line 2
split-file /usr/local/google/home/paulkirth/llvm-fork/clang/test/ClangScanDeps/strip-codegen-args.m /usr/local/google/home/paulkirth/llvm-fork/build/tools/clang/test/ClangScanDeps/Output/strip-codegen-args.m.tmp
# executed command: split-file /usr/local/google/home/paulkirth/llvm-fork/clang/test/ClangScanDeps/strip-codegen-args.m /usr/local/google/home/paulkirth/llvm-fork/build/tools/clang/test/ClangScanDeps/Output/strip-codegen-args.m.tmp
# RUN: at line 3
sed -e "s|DIR|/usr/local/google/home/paulkirth/llvm-fork/build/tools/clang/test/ClangScanDeps/Output/strip-codegen-args.m.tmp|g" /usr/local/google/home/paulkirth/llvm-fork/build/tools/clang/test/ClangScanDeps/Output/strip-codegen-args.m.tmp/cdb1.json.template > /usr/local/google/home/paulkirth/llvm-fork/build/tools/clang/test/ClangScanDeps/Output/strip-codegen-args.m.tmp/cdb1.json
# executed command: sed -e 's|DIR|/usr/local/google/home/paulkirth/llvm-fork/build/tools/clang/test/ClangScanDeps/Output/strip-codegen-args.m.tmp|g' /usr/local/google/home/paulkirth/llvm-fork/build/tools/clang/test/ClangScanDeps/Output/strip-codegen-args.m.tmp/cdb1.json.template
# RUN: at line 5
/usr/local/google/home/paulkirth/llvm-fork/build/bin/clang-scan-deps -compilation-database /usr/local/google/home/paulkirth/llvm-fork/build/tools/clang/test/ClangScanDeps/Output/strip-codegen-args.m.tmp/cdb1.json -format experimental-full > /usr/local/google/home/paulkirth/llvm-fork/build/tools/clang/test/ClangScanDeps/Output/strip-codegen-args.m.tmp/result1.txt
# executed command: /usr/local/google/home/paulkirth/llvm-fork/build/bin/clang-scan-deps -compilation-database /usr/local/google/home/paulkirth/llvm-fork/build/tools/clang/test/ClangScanDeps/Output/strip-codegen-args.m.tmp/cdb1.json -format experimental-full
# .---redirected output from '/usr/local/google/home/paulkirth/llvm-fork/build/tools/clang/test/ClangScanDeps/Output/strip-codegen-args.m.tmp/result1.txt'
# | {
# | "modules": [
# | {
# | "clang-module-deps": [],
# | "clang-modulemap-file": "/usr/local/google/home/paulkirth/llvm-fork/build/tools/clang/test/ClangScanDeps/Output/strip-codegen-args.m.tmp/modules/A/module.modulemap",
# | "command-line": [
# | "-cc1",
# | "-ferror-limit",
# | "19",
# | "-o",
# | "/usr/local/google/home/paulkirth/llvm-fork/build/tools/clang/test/ClangScanDeps/Output/strip-codegen-args.m.tmp/module-cache/D3YJALQADLMO6IDV7HKSHU6I6/A-D3YJALQADLMO6IDV7HKSHU6I6.pcm",
# | "-disable-free",
# | "-emit-module",
# | "-x",
# | "objective-c",
# | "/usr/local/google/home/paulkirth/llvm-fork/build/tools/clang/test/ClangScanDeps/Output/strip-codegen-args.m.tmp/modules/A/module.modulemap",
# | "-tune-cpu",
# | "generic",
# | "-target-cpu",
# | "x86-64",
# | "-triple",
# | "x86_64-unknown-linux-gnu",
# | "-fmodules-validate-system-headers",
# | "-fno-modules-prune-non-affecting-module-map-files",
# | "-resource-dir",
# |
# | ...
# `---data was truncated--------
# .---command stderr------------
# | YAML:17:3: error: Expected , between entries!
# | {
# | ^
# | ==================
# | WARNING: ThreadSanitizer: data race (pid=3573090)
# | Write of size 8 at 0x721c000009a0 by main thread:
# | #0 operator delete(void*, unsigned long, std::align_val_t) ../../../../../../llvm-llvm-project/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:190:3 (llvm+0x34e1a58) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #1 llvm::deallocate_buffer(void*, unsigned long, unsigned long) /usr/local/google/home/paulkirth/llvm-fork/llvm/lib/Support/MemAlloc.cpp:28:3 (llvm+0xc75a679) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #2 llvm::detail::UniqueFunctionBase::~UniqueFunctionBase() /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/ADT/FunctionExtras.h:285:7 (llvm+0x353f8be) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #3 std::__2::__tuple_leaf<0ul, llvm::unique_function, false>::~__tuple_leaf() /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/tuple:368:7 (llvm+0x353f8be)
# | #4 std::__2::tuple>::~tuple() /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/tuple:580:34 (llvm+0x353f8be)
# | #5 std::__2::__async_func>::~__async_func() /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/future:1834:22 (llvm+0x353f8be)
# | #6 std::__2::__deferred_assoc_state>>::~__deferred_assoc_state() /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/future:770:7 (llvm+0x353f8be)
# | #7 std::__2::__deferred_assoc_state>>::~__deferred_assoc_state() /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/future:770:7 (llvm+0x353f8be)
# | #8 clang_scan_deps_main(int, char**, llvm::ToolContext const&) /usr/local/google/home/paulkirth/llvm-fork/clang/tools/clang-scan-deps/ClangScanDeps.cpp:1157:7 (llvm+0x350e4d7) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #9 findTool(int, char**, char const*) /usr/local/google/home/paulkirth/llvm-fork/build/tools/llvm-driver/LLVMDriverTools.def:39:1 (llvm+0x3a06f57) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #10 main /usr/local/google/home/paulkirth/llvm-fork/llvm/tools/llvm-driver/llvm-driver.cpp:85:10 (llvm+0x3a06656) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# |
# | Previous read of size 8 at 0x721c000009a0 by thread T4:
# | #0 clang_scan_deps_main(int, char**, llvm::ToolContext const&)::$_3::operator()(clang::dependencies::DependencyScanningService&) const /usr/local/google/home/paulkirth/llvm-fork/clang/tools/clang-scan-deps/ClangScanDeps.cpp:988:35 (llvm+0x350f66e) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #1 clang_scan_deps_main(int, char**, llvm::ToolContext const&)::$_4::operator()() const /usr/local/google/home/paulkirth/llvm-fork/clang/tools/clang-scan-deps/ClangScanDeps.cpp:1157:47 (llvm+0x353fa56) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #2 void llvm::detail::UniqueFunctionBase::CallImpl(void*) /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/ADT/FunctionExtras.h:212:12 (llvm+0x353fa56)
# | #3 llvm::unique_function::operator()() /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/ADT/FunctionExtras.h:364:12 (llvm+0x353f966) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #4 std::__2::__invoke_result_impl>::type std::__2::__invoke[abi:nn220000]>(llvm::unique_function&&) /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/__type_traits/invoke.h:90:27 (llvm+0x353f966)
# | #5 void std::__2::__async_func>::__execute[abi:nn220000]<0ul>(std::__2::__integer_sequence) /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/future:1850:12 (llvm+0x353f966)
# | #6 std::__2::__async_func>::operator()[abi:nn220000]() /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/future:1845:51 (llvm+0x353f966)
# | #7 std::__2::__deferred_assoc_state>>::__execute() /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/future:821:5 (llvm+0x353f966)
# | #8 std::__2::shared_future::wait[abi:nn220000]() const /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/future:2034:55 (llvm+0x353f9a1) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #9 std::__2::shared_future llvm::ThreadPoolInterface::asyncImpl(llvm::unique_function, llvm::ThreadPoolTaskGroup*)::'lambda'()::operator()() const /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/ThreadPool.h:116:38 (llvm+0x353f9a1)
# | #10 void llvm::detail::UniqueFunctionBase::CallImpl llvm::ThreadPoolInterface::asyncImpl(llvm::unique_function, llvm::ThreadPoolTaskGroup*)::'lambda'()>(void*) /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/ADT/FunctionExtras.h:212:12 (llvm+0x353f9a1)
# | #11 llvm::unique_function::operator()() /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/ADT/FunctionExtras.h:364:12 (llvm+0xc7a1b6e) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #12 llvm::StdThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) /usr/local/google/home/paulkirth/llvm-fork/llvm/lib/Support/ThreadPool.cpp:119:5 (llvm+0xc7a1b6e)
# | #13 llvm::StdThreadPool::grow(int)::$_0::operator()() const /usr/local/google/home/paulkirth/llvm-fork/llvm/lib/Support/ThreadPool.cpp:62:9 (llvm+0xc7a51b7) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #14 auto void llvm::thread::GenericThreadProxy>(void*)::'lambda'(auto&&, auto&&...)::operator()(auto&&, auto&&...) const /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/thread.h:46:11 (llvm+0xc7a51b7)
# | #15 std::__2::__invoke_result_impl>(void*)::'lambda'(auto&&, auto&&...), llvm::StdThreadPool::grow(int)::$_0&>::type std::__2::__invoke[abi:nn220000]>(void*)::'lambda'(auto&&, auto&&...), llvm::StdThreadPool::grow(int)::$_0&>(void llvm::thread::GenericThreadProxy>(void*)::'lambda'(auto&&, auto&&...)&&, llvm::StdThreadPool::grow(int)::$_0&) /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/__type_traits/invoke.h:90:27 (llvm+0xc7a51b7)
# | #16 decltype(auto) std::__2::__apply_tuple_impl[abi:nn220000]>(void*)::'lambda'(auto&&, auto&&...), std::__2::tuple&, 0ul>(auto&&, std::__2::tuple&, std::__2::integer_sequence) /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/tuple:1401:5 (llvm+0xc7a51b7)
# | #17 decltype(auto) std::__2::apply[abi:nn220000]>(void*)::'lambda'(auto&&, auto&&...), std::__2::tuple&>(auto&&, std::__2::tuple&) /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/tuple:1405:5 (llvm+0xc7a51b7)
# | #18 void llvm::thread::GenericThreadProxy>(void*) /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/thread.h:44:5 (llvm+0xc7a51b7)
# | #19 void* llvm::thread::ThreadProxy>(void*) /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/thread.h:62:5 (llvm+0xc7a51b7)
# |
# | Thread T4 'llvm-worker-3' (tid=3573363, running) created by main thread at:
# | #0 pthread_create ../../../../../../llvm-llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1075:3 (llvm+0x345f11a) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #1 llvm::llvm_execute_on_thread_impl(void* (*)(void*), void*, std::__2::optional) /usr/local/google/home/paulkirth/llvm-fork/llvm/lib/Support/Unix/Threading.inc:95:17 (llvm+0xc8324b5) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #2 llvm::thread::thread(std::__2::optional, llvm::StdThreadPool::grow(int)::$_0&) /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/thread.h:138:12 (llvm+0xc7a124f) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #3 llvm::thread::thread(llvm::StdThreadPool::grow(int)::$_0&&) /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/thread.h:85:9 (llvm+0xc7a124f)
# | #4 void std::__2::allocator::construct[abi:nn220000](llvm::thread*, llvm::StdThreadPool::grow(int)::$_0&&) /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/__memory/allocator.h:153:24 (llvm+0xc7a124f)
# | #5 void std::__2::allocator_traits>::construct[abi:nn220000](std::__2::allocator&, llvm::thread*, llvm::StdThreadPool::grow(int)::$_0&&) /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/__memory/allocator_traits.h:296:9 (llvm+0xc7a124f)
# | #6 void std::__2::vector>::__emplace_back_assume_capacity[abi:nn220000](llvm::StdThreadPool::grow(int)::$_0&&) /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/__vector/vector.h:483:5 (llvm+0xc7a124f)
# | #7 llvm::thread& std::__2::vector>::emplace_back(llvm::StdThreadPool::grow(int)::$_0&&) /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/__vector/vector.h:1156:5 (llvm+0xc7a124f)
# | #8 llvm::StdThreadPool::grow(int) /usr/local/google/home/paulkirth/llvm-fork/llvm/lib/Support/ThreadPool.cpp:51:13 (llvm+0xc7a124f)
# | #9 llvm::StdThreadPool::asyncEnqueue(llvm::unique_function, llvm::ThreadPoolTaskGroup*) /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/ThreadPool.h:176:5 (llvm+0xc7a3718) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #10 std::__2::shared_future llvm::ThreadPoolInterface::asyncImpl(llvm::unique_function, llvm::ThreadPoolTaskGroup*) /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/ThreadPool.h:116:5 (llvm+0x353e9e5) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #11 std::__2::shared_future llvm::ThreadPoolInterface::async(clang_scan_deps_main(int, char**, llvm::ToolContext const&)::$_4&&) /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/ThreadPool.h:98:12 (llvm+0x350e59b) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #12 clang_scan_deps_main(int, char**, llvm::ToolContext const&) /usr/local/google/home/paulkirth/llvm-fork/clang/tools/clang-scan-deps/ClangScanDeps.cpp:1157:12 (llvm+0x350e59b)
# | #13 findTool(int, char**, char const*) /usr/local/google/home/paulkirth/llvm-fork/build/tools/llvm-driver/LLVMDriverTools.def:39:1 (llvm+0x3a06f57) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #14 main /usr/local/google/home/paulkirth/llvm-fork/llvm/tools/llvm-driver/llvm-driver.cpp:85:10 (llvm+0x3a06656) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# |
# | SUMMARY: ThreadSanitizer: data race /usr/local/google/home/paulkirth/llvm-fork/llvm/lib/Support/MemAlloc.cpp:28:3 in llvm::deallocate_buffer(void*, unsigned long, unsigned long)
# | ==================
# | ==================
# | WARNING: ThreadSanitizer: data race (pid=3573090)
# | Write of size 8 at 0x722800000680 by main thread:
# | #0 pthread_cond_destroy ../../../../../../llvm-llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1367:3 (llvm+0x346090a) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #1 std::__2::__assoc_sub_state::~__assoc_sub_state() /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/future:521:33 (llvm+0x353f8e0) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #2 std::__2::__deferred_assoc_state>>::~__deferred_assoc_state() /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/future:770:7 (llvm+0x353f8e0)
# | #3 std::__2::__deferred_assoc_state>>::~__deferred_assoc_state() /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/future:770:7 (llvm+0x353f8e0)
# | #4 clang_scan_deps_main(int, char**, llvm::ToolContext const&) /usr/local/google/home/paulkirth/llvm-fork/clang/tools/clang-scan-deps/ClangScanDeps.cpp:1157:7 (llvm+0x350e4d7) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #5 findTool(int, char**, char const*) /usr/local/google/home/paulkirth/llvm-fork/build/tools/llvm-driver/LLVMDriverTools.def:39:1 (llvm+0x3a06f57) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #6 main /usr/local/google/home/paulkirth/llvm-fork/llvm/tools/llvm-driver/llvm-driver.cpp:85:10 (llvm+0x3a06656) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# |
# | Previous read of size 8 at 0x722800000680 by thread T4 (mutexes: write M0):
# | #0 pthread_cond_broadcast ../../../../../../llvm-llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1360:3 (llvm+0x3460778) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #1 std::__2::__assoc_sub_state::set_value() ../../../../../../llvm-llvm-project/libcxx/src/future.cpp:67:9 (llvm+0xc952f8b) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #2 std::__2::shared_future::wait[abi:nn220000]() const /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/future:2034:55 (llvm+0x353f9a1) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #3 std::__2::shared_future llvm::ThreadPoolInterface::asyncImpl(llvm::unique_function, llvm::ThreadPoolTaskGroup*)::'lambda'()::operator()() const /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/ThreadPool.h:116:38 (llvm+0x353f9a1)
# | #4 void llvm::detail::UniqueFunctionBase::CallImpl llvm::ThreadPoolInterface::asyncImpl(llvm::unique_function, llvm::ThreadPoolTaskGroup*)::'lambda'()>(void*) /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/ADT/FunctionExtras.h:212:12 (llvm+0x353f9a1)
# | #5 llvm::unique_function::operator()() /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/ADT/FunctionExtras.h:364:12 (llvm+0xc7a1b6e) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #6 llvm::StdThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) /usr/local/google/home/paulkirth/llvm-fork/llvm/lib/Support/ThreadPool.cpp:119:5 (llvm+0xc7a1b6e)
# | #7 llvm::StdThreadPool::grow(int)::$_0::operator()() const /usr/local/google/home/paulkirth/llvm-fork/llvm/lib/Support/ThreadPool.cpp:62:9 (llvm+0xc7a51b7) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #8 auto void llvm::thread::GenericThreadProxy>(void*)::'lambda'(auto&&, auto&&...)::operator()(auto&&, auto&&...) const /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/thread.h:46:11 (llvm+0xc7a51b7)
# | #9 std::__2::__invoke_result_impl>(void*)::'lambda'(auto&&, auto&&...), llvm::StdThreadPool::grow(int)::$_0&>::type std::__2::__invoke[abi:nn220000]>(void*)::'lambda'(auto&&, auto&&...), llvm::StdThreadPool::grow(int)::$_0&>(void llvm::thread::GenericThreadProxy>(void*)::'lambda'(auto&&, auto&&...)&&, llvm::StdThreadPool::grow(int)::$_0&) /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/__type_traits/invoke.h:90:27 (llvm+0xc7a51b7)
# | #10 decltype(auto) std::__2::__apply_tuple_impl[abi:nn220000]>(void*)::'lambda'(auto&&, auto&&...), std::__2::tuple&, 0ul>(auto&&, std::__2::tuple&, std::__2::integer_sequence) /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/tuple:1401:5 (llvm+0xc7a51b7)
# | #11 decltype(auto) std::__2::apply[abi:nn220000]>(void*)::'lambda'(auto&&, auto&&...), std::__2::tuple&>(auto&&, std::__2::tuple&) /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/tuple:1405:5 (llvm+0xc7a51b7)
# | #12 void llvm::thread::GenericThreadProxy>(void*) /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/thread.h:44:5 (llvm+0xc7a51b7)
# | #13 void* llvm::thread::ThreadProxy>(void*) /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/thread.h:62:5 (llvm+0xc7a51b7)
# |
# | Location is heap block of size 152 at 0x722800000640 allocated by main thread:
# | #0 operator new(unsigned long) ../../../../../../llvm-llvm-project/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:64:3 (llvm+0x34e0a02) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #1 std::__2::future std::__2::__make_deferred_assoc_state[abi:nn220000]>>(std::__2::__async_func>&&) /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/future:1810:7 (llvm+0x353f1ea) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #2 std::__2::future)>::type> std::__2::async[abi:nn220000]>(std::__2::launch, llvm::unique_function&&) /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/future:1878:12 (llvm+0x353ee66) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #3 std::__2::shared_future llvm::ThreadPoolInterface::asyncImpl(llvm::unique_function, llvm::ThreadPoolTaskGroup*) /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/ThreadPool.h:115:19 (llvm+0x353e93a) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #4 std::__2::shared_future llvm::ThreadPoolInterface::async(clang_scan_deps_main(int, char**, llvm::ToolContext const&)::$_4&&) /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/ThreadPool.h:98:12 (llvm+0x350e59b) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #5 clang_scan_deps_main(int, char**, llvm::ToolContext const&) /usr/local/google/home/paulkirth/llvm-fork/clang/tools/clang-scan-deps/ClangScanDeps.cpp:1157:12 (llvm+0x350e59b)
# | #6 findTool(int, char**, char const*) /usr/local/google/home/paulkirth/llvm-fork/build/tools/llvm-driver/LLVMDriverTools.def:39:1 (llvm+0x3a06f57) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #7 main /usr/local/google/home/paulkirth/llvm-fork/llvm/tools/llvm-driver/llvm-driver.cpp:85:10 (llvm+0x3a06656) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# |
# | Mutex M0 (0x722800000658) created at:
# | #0 pthread_mutex_lock ../../../../../../llvm-llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1405:3 (llvm+0x3460db7) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #1 std::__2::__libcpp_mutex_lock[abi:ne220000](pthread_mutex_t*) /b/s/w/ir/x/w/llvm-llvm-project/libcxx/include/__thread/support/pthread.h:95:10 (llvm+0xc953605) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #2 std::__2::mutex::lock() ../../../../../../llvm-llvm-project/libcxx/src/mutex.cpp:30:12 (llvm+0xc953605)
# | #3 std::__2::future)>::type> std::__2::async[abi:nn220000]>(std::__2::launch, llvm::unique_function&&) /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/future:1878:12 (llvm+0x353ee66) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #4 std::__2::shared_future llvm::ThreadPoolInterface::asyncImpl(llvm::unique_function, llvm::ThreadPoolTaskGroup*) /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/ThreadPool.h:115:19 (llvm+0x353e93a) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #5 std::__2::shared_future llvm::ThreadPoolInterface::async(clang_scan_deps_main(int, char**, llvm::ToolContext const&)::$_4&&) /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/ThreadPool.h:98:12 (llvm+0x350e59b) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #6 clang_scan_deps_main(int, char**, llvm::ToolContext const&) /usr/local/google/home/paulkirth/llvm-fork/clang/tools/clang-scan-deps/ClangScanDeps.cpp:1157:12 (llvm+0x350e59b)
# | #7 findTool(int, char**, char const*) /usr/local/google/home/paulkirth/llvm-fork/build/tools/llvm-driver/LLVMDriverTools.def:39:1 (llvm+0x3a06f57) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #8 main /usr/local/google/home/paulkirth/llvm-fork/llvm/tools/llvm-driver/llvm-driver.cpp:85:10 (llvm+0x3a06656) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# |
# | Thread T4 'llvm-worker-3' (tid=3573363, running) created by main thread at:
# | #0 pthread_create ../../../../../../llvm-llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1075:3 (llvm+0x345f11a) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #1 llvm::llvm_execute_on_thread_impl(void* (*)(void*), void*, std::__2::optional) /usr/local/google/home/paulkirth/llvm-fork/llvm/lib/Support/Unix/Threading.inc:95:17 (llvm+0xc8324b5) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #2 llvm::thread::thread(std::__2::optional, llvm::StdThreadPool::grow(int)::$_0&) /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/thread.h:138:12 (llvm+0xc7a124f) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #3 llvm::thread::thread(llvm::StdThreadPool::grow(int)::$_0&&) /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/thread.h:85:9 (llvm+0xc7a124f)
# | #4 void std::__2::allocator::construct[abi:nn220000](llvm::thread*, llvm::StdThreadPool::grow(int)::$_0&&) /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/__memory/allocator.h:153:24 (llvm+0xc7a124f)
# | #5 void std::__2::allocator_traits>::construct[abi:nn220000](std::__2::allocator&, llvm::thread*, llvm::StdThreadPool::grow(int)::$_0&&) /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/__memory/allocator_traits.h:296:9 (llvm+0xc7a124f)
# | #6 void std::__2::vector>::__emplace_back_assume_capacity[abi:nn220000](llvm::StdThreadPool::grow(int)::$_0&&) /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/__vector/vector.h:483:5 (llvm+0xc7a124f)
# | #7 llvm::thread& std::__2::vector>::emplace_back(llvm::StdThreadPool::grow(int)::$_0&&) /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/__vector/vector.h:1156:5 (llvm+0xc7a124f)
# | #8 llvm::StdThreadPool::grow(int) /usr/local/google/home/paulkirth/llvm-fork/llvm/lib/Support/ThreadPool.cpp:51:13 (llvm+0xc7a124f)
# | #9 llvm::StdThreadPool::asyncEnqueue(llvm::unique_function, llvm::ThreadPoolTaskGroup*) /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/ThreadPool.h:176:5 (llvm+0xc7a3718) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #10 std::__2::shared_future llvm::ThreadPoolInterface::asyncImpl(llvm::unique_function, llvm::ThreadPoolTaskGroup*) /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/ThreadPool.h:116:5 (llvm+0x353e9e5) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #11 std::__2::shared_future llvm::ThreadPoolInterface::async(clang_scan_deps_main(int, char**, llvm::ToolContext const&)::$_4&&) /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/ThreadPool.h:98:12 (llvm+0x350e59b) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #12 clang_scan_deps_main(int, char**, llvm::ToolContext const&) /usr/local/google/home/paulkirth/llvm-fork/clang/tools/clang-scan-deps/ClangScanDeps.cpp:1157:12 (llvm+0x350e59b)
# | #13 findTool(int, char**, char const*) /usr/local/google/home/paulkirth/llvm-fork/build/tools/llvm-driver/LLVMDriverTools.def:39:1 (llvm+0x3a06f57) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #14 main /usr/local/google/home/paulkirth/llvm-fork/llvm/tools/llvm-driver/llvm-driver.cpp:85:10 (llvm+0x3a06656) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# |
# | SUMMARY: ThreadSanitizer: data race /usr/local/google/home/paulkirth/llvm-fork/clang/tools/clang-scan-deps/ClangScanDeps.cpp:1157:7 in clang_scan_deps_main(int, char**, llvm::ToolContext const&)
# | ==================
# | ==================
# | WARNING: ThreadSanitizer: data race (pid=3573090)
# | Write of size 8 at 0x722800000658 by main thread:
# | #0 operator delete(void*, unsigned long) ../../../../../../llvm-llvm-project/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:150:3 (llvm+0x34e1518) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #1 std::__2::__deferred_assoc_state>>::~__deferred_assoc_state() /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/future:770:7 (llvm+0x353f8fe) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #2 clang_scan_deps_main(int, char**, llvm::ToolContext const&) /usr/local/google/home/paulkirth/llvm-fork/clang/tools/clang-scan-deps/ClangScanDeps.cpp:1157:7 (llvm+0x350e4d7) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #3 findTool(int, char**, char const*) /usr/local/google/home/paulkirth/llvm-fork/build/tools/llvm-driver/LLVMDriverTools.def:39:1 (llvm+0x3a06f57) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #4 main /usr/local/google/home/paulkirth/llvm-fork/llvm/tools/llvm-driver/llvm-driver.cpp:85:10 (llvm+0x3a06656) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# |
# | Previous atomic read of size 1 at 0x722800000658 by thread T4:
# | #0 pthread_mutex_lock ../../../../../../llvm-llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1405:3 (llvm+0x3460db7) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #1 std::__2::__libcpp_mutex_lock[abi:ne220000](pthread_mutex_t*) /b/s/w/ir/x/w/llvm-llvm-project/libcxx/include/__thread/support/pthread.h:95:10 (llvm+0xc953605) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #2 std::__2::mutex::lock() ../../../../../../llvm-llvm-project/libcxx/src/mutex.cpp:30:12 (llvm+0xc953605)
# | #3 std::__2::shared_future::wait[abi:nn220000]() const /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/future:2034:55 (llvm+0x353f9a1) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #4 std::__2::shared_future llvm::ThreadPoolInterface::asyncImpl(llvm::unique_function, llvm::ThreadPoolTaskGroup*)::'lambda'()::operator()() const /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/ThreadPool.h:116:38 (llvm+0x353f9a1)
# | #5 void llvm::detail::UniqueFunctionBase::CallImpl llvm::ThreadPoolInterface::asyncImpl(llvm::unique_function, llvm::ThreadPoolTaskGroup*)::'lambda'()>(void*) /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/ADT/FunctionExtras.h:212:12 (llvm+0x353f9a1)
# | #6 llvm::unique_function::operator()() /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/ADT/FunctionExtras.h:364:12 (llvm+0xc7a1b6e) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #7 llvm::StdThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) /usr/local/google/home/paulkirth/llvm-fork/llvm/lib/Support/ThreadPool.cpp:119:5 (llvm+0xc7a1b6e)
# | #8 llvm::StdThreadPool::grow(int)::$_0::operator()() const /usr/local/google/home/paulkirth/llvm-fork/llvm/lib/Support/ThreadPool.cpp:62:9 (llvm+0xc7a51b7) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #9 auto void llvm::thread::GenericThreadProxy>(void*)::'lambda'(auto&&, auto&&...)::operator()(auto&&, auto&&...) const /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/thread.h:46:11 (llvm+0xc7a51b7)
# | #10 std::__2::__invoke_result_impl>(void*)::'lambda'(auto&&, auto&&...), llvm::StdThreadPool::grow(int)::$_0&>::type std::__2::__invoke[abi:nn220000]>(void*)::'lambda'(auto&&, auto&&...), llvm::StdThreadPool::grow(int)::$_0&>(void llvm::thread::GenericThreadProxy>(void*)::'lambda'(auto&&, auto&&...)&&, llvm::StdThreadPool::grow(int)::$_0&) /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/__type_traits/invoke.h:90:27 (llvm+0xc7a51b7)
# | #11 decltype(auto) std::__2::__apply_tuple_impl[abi:nn220000]>(void*)::'lambda'(auto&&, auto&&...), std::__2::tuple&, 0ul>(auto&&, std::__2::tuple&, std::__2::integer_sequence) /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/tuple:1401:5 (llvm+0xc7a51b7)
# | #12 decltype(auto) std::__2::apply[abi:nn220000]>(void*)::'lambda'(auto&&, auto&&...), std::__2::tuple&>(auto&&, std::__2::tuple&) /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/tuple:1405:5 (llvm+0xc7a51b7)
# | #13 void llvm::thread::GenericThreadProxy>(void*) /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/thread.h:44:5 (llvm+0xc7a51b7)
# | #14 void* llvm::thread::ThreadProxy>(void*) /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/thread.h:62:5 (llvm+0xc7a51b7)
# |
# | Thread T4 'llvm-worker-3' (tid=3573363, running) created by main thread at:
# | #0 pthread_create ../../../../../../llvm-llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1075:3 (llvm+0x345f11a) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #1 llvm::llvm_execute_on_thread_impl(void* (*)(void*), void*, std::__2::optional) /usr/local/google/home/paulkirth/llvm-fork/llvm/lib/Support/Unix/Threading.inc:95:17 (llvm+0xc8324b5) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #2 llvm::thread::thread(std::__2::optional, llvm::StdThreadPool::grow(int)::$_0&) /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/thread.h:138:12 (llvm+0xc7a124f) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #3 llvm::thread::thread(llvm::StdThreadPool::grow(int)::$_0&&) /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/thread.h:85:9 (llvm+0xc7a124f)
# | #4 void std::__2::allocator::construct[abi:nn220000](llvm::thread*, llvm::StdThreadPool::grow(int)::$_0&&) /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/__memory/allocator.h:153:24 (llvm+0xc7a124f)
# | #5 void std::__2::allocator_traits>::construct[abi:nn220000](std::__2::allocator&, llvm::thread*, llvm::StdThreadPool::grow(int)::$_0&&) /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/__memory/allocator_traits.h:296:9 (llvm+0xc7a124f)
# | #6 void std::__2::vector>::__emplace_back_assume_capacity[abi:nn220000](llvm::StdThreadPool::grow(int)::$_0&&) /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/__vector/vector.h:483:5 (llvm+0xc7a124f)
# | #7 llvm::thread& std::__2::vector>::emplace_back(llvm::StdThreadPool::grow(int)::$_0&&) /usr/local/google/home/paulkirth/fuchsia/prebuilt/third_party/clang/linux-x64/bin/../include/c++/v1/__vector/vector.h:1156:5 (llvm+0xc7a124f)
# | #8 llvm::StdThreadPool::grow(int) /usr/local/google/home/paulkirth/llvm-fork/llvm/lib/Support/ThreadPool.cpp:51:13 (llvm+0xc7a124f)
# | #9 llvm::StdThreadPool::asyncEnqueue(llvm::unique_function, llvm::ThreadPoolTaskGroup*) /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/ThreadPool.h:176:5 (llvm+0xc7a3718) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #10 std::__2::shared_future llvm::ThreadPoolInterface::asyncImpl(llvm::unique_function, llvm::ThreadPoolTaskGroup*) /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/ThreadPool.h:116:5 (llvm+0x353e9e5) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #11 std::__2::shared_future llvm::ThreadPoolInterface::async(clang_scan_deps_main(int, char**, llvm::ToolContext const&)::$_4&&) /usr/local/google/home/paulkirth/llvm-fork/llvm/include/llvm/Support/ThreadPool.h:98:12 (llvm+0x350e59b) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #12 clang_scan_deps_main(int, char**, llvm::ToolContext const&) /usr/local/google/home/paulkirth/llvm-fork/clang/tools/clang-scan-deps/ClangScanDeps.cpp:1157:12 (llvm+0x350e59b)
# | #13 findTool(int, char**, char const*) /usr/local/google/home/paulkirth/llvm-fork/build/tools/llvm-driver/LLVMDriverTools.def:39:1 (llvm+0x3a06f57) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# | #14 main /usr/local/google/home/paulkirth/llvm-fork/llvm/tools/llvm-driver/llvm-driver.cpp:85:10 (llvm+0x3a06656) (BuildId: 8fac594a806d2a8a01a2101c57cdb487de731a91)
# |
# | SUMMARY: ThreadSanitizer: data race /usr/local/google/home/paulkirth/llvm-fork/clang/tools/clang-scan-deps/ClangScanDeps.cpp:1157:7 in clang_scan_deps_main(int, char**, llvm::ToolContext const&)
# | ==================
# | ThreadSanitizer: reported 3 warnings
# `-----------------------------
# error: command failed with exit status: 66

```

I checked out LLVM back from September, and it still seems to fail, which indicates to me this would have been the state of affairs for some time. To cross reference, I also tried rebuilding the toolchain with a clang from back in Sept and the failure persists, so I don't necessarily think its an issue with the compiler itself (miscompile) or the runtimes (unless this is a known issue/limitation).

Reproduce the failures with check-clang and check-llvm under TSan, starting with clang/test/ClangScanDeps/strip-codegen-args.m. Read clang/tools/clang-scan-deps/ClangScanDeps.cpp around lines 988 and 1157, then compare the reported race with the ThreadPool paths in llvm/include/llvm/Support/ThreadPool.h and llvm/lib/Support/ThreadPool.cpp. Done means the affected Clang, LLVM, and unit tests pass under TSan without race reports.

Written by the indexing model from the issue text.