ClickHouse / ClickHouse/ClickHouse
Dictionary error after upgrade to 23.9.5.29 - Cannot schedule a task: cannot allocate thread
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
Hello! I'm faced with error after upgrade from 23.2.2 to 23.9.5.29:
```
2023.11.22 11:25:12.181056 [ 2383151 ] {4fe8e91d-10c9-4ee3-9350-c88ccda39a79} TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=5, jobs=5): while executing 'FUNCTION dictGetFloat32('dict.chrt_price' : 13, 'price' :: 14, chrt_id : 1) -> dictGetFloat32('dict.chrt_price', 'price', chrt_id) Float32 : 12'. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):
0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c73d137 in /usr/bin/clickhouse
1. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String const&, unsigned long&&, unsigned long&) @ 0x000000000c825666 in /usr/bin/clickhouse
2. void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x000000000c82718e in /usr/bin/clickhouse
3. void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool) @ 0x000000000c82161e in /usr/bin/clickhouse
4. DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector, std::allocator>>&&, std::vector, std::allocator>>&&, std::shared_ptr, std::shared_ptr const&, DB::KeyCondition const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, Poco::Logger*, unsigned long, std::vector>&, bool) @ 0x0000000012ebc1c7 in /usr/bin/clickhouse
5. DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector, std::allocator>>, std::vector, std::allocator>>, std::shared_ptr const&, std::shared_ptr const&, DB::SelectQueryInfo const&, std::shared_ptr, unsigned long, std::shared_ptr, std::equal_to, std::allocator>>>, DB::MergeTreeData const&, std::vector> const&, bool, Poco::Logger*, std::optional&) @ 0x00000000137808fb in /usr/bin/clickhouse
6. DB::ReadFromMergeTree::selectRangesToRead(std::vector, std::allocator>>, std::vector, std::allocator>>, std::shared_ptr const&, DB::ActionDAGNodes const&, std::shared_ptr const&, std::shared_ptr const&, DB::SelectQueryInfo const&, std::shared_ptr, unsigned long, std::shared_ptr, std::equal_to, std::allocator>>>, DB::MergeTreeData const&, std::vector> const&, bool, Poco::Logger*, std::optional&) @ 0x000000001377bb27 in /usr/bin/clickhouse
7. DB::ReadFromMergeTree::selectRangesToRead(std::vector, std::allocator>>, std::vector, std::allocator>>) const @ 0x000000001377b7d0 in /usr/bin/clickhouse
8. DB::ReadFromMergeTree::getAnalysisResult() const @ 0x0000000013783bfa in /usr/bin/clickhouse
9. DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x0000000013785d47 in /usr/bin/clickhouse
10. DB::ISourceStep::updatePipeline(std::vector>, std::allocator>>>, DB::BuildQueryPipelineSettings const&) @ 0x0000000013748761 in /usr/bin/clickhouse
11. DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x0000000013760467 in /usr/bin/clickhouse
12. DB::InterpreterSelectWithUnionQuery::execute() @ 0x0000000012139e20 in /usr/bin/clickhouse
13. DB::executeQueryImpl(char const*, char const*, std::shared_ptr, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x0000000012491556 in /usr/bin/clickhouse
14. DB::executeQuery(String const&, std::shared_ptr, bool, DB::QueryProcessingStage::Enum) @ 0x000000001248cc5c in /usr/bin/clickhouse
15. DB::ClickHouseDictionarySource::createStreamForQuery(String const&) @ 0x000000000f4a3517 in /usr/bin/clickhouse
16. DB::ClickHouseDictionarySource::loadAll() @ 0x000000000f4a3370 in /usr/bin/clickhouse
17. DB::HashedDictionary<(DB::DictionaryKeyType)0, false, false>::loadData() @ 0x000000000f58a75f in /usr/bin/clickhouse
18. DB::HashedDictionary<(DB::DictionaryKeyType)0, false, false>::HashedDictionary(DB::StorageID const&, DB::DictionaryStructure const&, std::shared_ptr, DB::HashedDictionaryConfiguration const&, std::shared_ptr) @ 0x000000000f5893dd in /usr/bin/clickhouse
19. DB::registerDictionaryHashed(DB::DictionaryFactory&)::$_4::operator()(String const&, DB::DictionaryStructure const&, Poco::Util::AbstractConfiguration const&, String const&, std::shared_ptr, DB::DictionaryKeyType, bool) const @ 0x000000000f8e48de in /usr/bin/clickhouse
20. std::unique_ptr> std::__function::__policy_invoker> (String const&, DB::DictionaryStructure const&, Poco::Util::AbstractConfiguration const&, String const&, std::shared_ptr, std::shared_ptr, bool)>::__call_impl> (String const&, DB::DictionaryStructure const&, Poco::Util::AbstractConfiguration const&, String const&, std::shared_ptr, std::shared_ptr, bool)>>(std::__function::__policy_storage const*, String const&, DB::DictionaryStructure const&, Poco::Util::AbstractConfiguration const&, String const&, std::shared_ptr&&, std::shared_ptr&&, bool) @ 0x000000000f8e33e5 in /usr/bin/clickhouse
21. DB::DictionaryFactory::create(String const&, Poco::Util::AbstractConfiguration const&, String const&, std::shared_ptr, bool) const @ 0x0000000010d3b87d in /usr/bin/clickhouse
22. DB::ExternalDictionariesLoader::create(String const&, Poco::Util::AbstractConfiguration const&, String const&, String const&) const @ 0x0000000011cb561c in /usr/bin/clickhouse
23. DB::ExternalLoader::LoadingDispatcher::doLoading(String const&, unsigned long, bool, unsigned long, bool, std::shared_ptr) @ 0x0000000011cc0897 in /usr/bin/clickhouse
24. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl), DB::ExternalLoader::LoadingDispatcher*, String&, unsigned long&, bool&, unsigned long&, bool, std::shared_ptr>(void (DB::ExternalLoader::LoadingDispatcher::*&&)(String const&, unsigned long, bool, unsigned long, bool, std::shared_ptr), DB::ExternalLoader::LoadingDispatcher*&&, String&, unsigned long&, bool&, unsigned long&, bool&&, std::shared_ptr&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000011cc595f in /usr/bin/clickhouse
25. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c826567 in /usr/bin/clickhouse
26. start_thread @ 0x0000000000007ea7 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
27. ? @ 0x00000000000fba2f in /usr/lib/x86_64-linux-gnu/libc-2.31.so
```
Query:
```
select dictGetFloat32('dict.chrt_price', 'price', 1)
```
All queries which contain this dictionary failed with this exception (other queries - ok).
Also this happened only at specific node (cluster with 5 nodes).
The problem was fixed by drop and create dictionary on the problematic node. Dictionary source= Clickhouse, layout=HASHED()
Upgrade steps:
```
1. sudo apt-get install clickhouse-server=23.9.5.29 clickhouse-client=23.9.5.29 clickhouse-common-static=23.9.5.29
2. service clickhouse-server restart
```
Any idea why it happened and how to avoid this problem at the future updates?
Contributor guide
Assessment
This issue has not been assessed yet.