ClickHouse / ClickHouse/ClickHouse

Dictionary error after upgrade to 23.9.5.29 - Cannot schedule a task: cannot allocate thread

Open
#57,094 1 comment 0 reactions 0 assignees View on GitHub
comp-dictionary external question unexpected behaviour
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

Hello! I'm faced with error after upgrade from 23.2.2 to 23.9.5.29:
```
2023.11.22 11:25:12.181056 [ 2383151 ] {4fe8e91d-10c9-4ee3-9350-c88ccda39a79} TCPHandler: Code: 439. DB::Exception: Cannot schedule a task: cannot allocate thread (threads=5, jobs=5): while executing 'FUNCTION dictGetFloat32('dict.chrt_price' : 13, 'price' :: 14, chrt_id : 1) -> dictGetFloat32('dict.chrt_price', 'price', chrt_id) Float32 : 12'. (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c73d137 in /usr/bin/clickhouse
1. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String const&, unsigned long&&, unsigned long&) @ 0x000000000c825666 in /usr/bin/clickhouse
2. void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x000000000c82718e in /usr/bin/clickhouse
3. void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool) @ 0x000000000c82161e in /usr/bin/clickhouse
4. DB::MergeTreeDataSelectExecutor::filterPartsByPrimaryKeyAndSkipIndexes(std::vector, std::allocator>>&&, std::vector, std::allocator>>&&, std::shared_ptr, std::shared_ptr const&, DB::KeyCondition const&, DB::UsefulSkipIndexes const&, DB::MergeTreeReaderSettings const&, Poco::Logger*, unsigned long, std::vector>&, bool) @ 0x0000000012ebc1c7 in /usr/bin/clickhouse
5. DB::ReadFromMergeTree::selectRangesToReadImpl(std::vector, std::allocator>>, std::vector, std::allocator>>, std::shared_ptr const&, std::shared_ptr const&, DB::SelectQueryInfo const&, std::shared_ptr, unsigned long, std::shared_ptr, std::equal_to, std::allocator>>>, DB::MergeTreeData const&, std::vector> const&, bool, Poco::Logger*, std::optional&) @ 0x00000000137808fb in /usr/bin/clickhouse
6. DB::ReadFromMergeTree::selectRangesToRead(std::vector, std::allocator>>, std::vector, std::allocator>>, std::shared_ptr const&, DB::ActionDAGNodes const&, std::shared_ptr const&, std::shared_ptr const&, DB::SelectQueryInfo const&, std::shared_ptr, unsigned long, std::shared_ptr, std::equal_to, std::allocator>>>, DB::MergeTreeData const&, std::vector> const&, bool, Poco::Logger*, std::optional&) @ 0x000000001377bb27 in /usr/bin/clickhouse
7. DB::ReadFromMergeTree::selectRangesToRead(std::vector, std::allocator>>, std::vector, std::allocator>>) const @ 0x000000001377b7d0 in /usr/bin/clickhouse
8. DB::ReadFromMergeTree::getAnalysisResult() const @ 0x0000000013783bfa in /usr/bin/clickhouse
9. DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x0000000013785d47 in /usr/bin/clickhouse

10. DB::ISourceStep::updatePipeline(std::vector>, std::allocator>>>, DB::BuildQueryPipelineSettings const&) @ 0x0000000013748761 in /usr/bin/clickhouse
11. DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x0000000013760467 in /usr/bin/clickhouse
12. DB::InterpreterSelectWithUnionQuery::execute() @ 0x0000000012139e20 in /usr/bin/clickhouse
13. DB::executeQueryImpl(char const*, char const*, std::shared_ptr, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x0000000012491556 in /usr/bin/clickhouse
14. DB::executeQuery(String const&, std::shared_ptr, bool, DB::QueryProcessingStage::Enum) @ 0x000000001248cc5c in /usr/bin/clickhouse
15. DB::ClickHouseDictionarySource::createStreamForQuery(String const&) @ 0x000000000f4a3517 in /usr/bin/clickhouse
16. DB::ClickHouseDictionarySource::loadAll() @ 0x000000000f4a3370 in /usr/bin/clickhouse
17. DB::HashedDictionary<(DB::DictionaryKeyType)0, false, false>::loadData() @ 0x000000000f58a75f in /usr/bin/clickhouse
18. DB::HashedDictionary<(DB::DictionaryKeyType)0, false, false>::HashedDictionary(DB::StorageID const&, DB::DictionaryStructure const&, std::shared_ptr, DB::HashedDictionaryConfiguration const&, std::shared_ptr) @ 0x000000000f5893dd in /usr/bin/clickhouse
19. DB::registerDictionaryHashed(DB::DictionaryFactory&)::$_4::operator()(String const&, DB::DictionaryStructure const&, Poco::Util::AbstractConfiguration const&, String const&, std::shared_ptr, DB::DictionaryKeyType, bool) const @ 0x000000000f8e48de in /usr/bin/clickhouse
20. std::unique_ptr> std::__function::__policy_invoker> (String const&, DB::DictionaryStructure const&, Poco::Util::AbstractConfiguration const&, String const&, std::shared_ptr, std::shared_ptr, bool)>::__call_impl> (String const&, DB::DictionaryStructure const&, Poco::Util::AbstractConfiguration const&, String const&, std::shared_ptr, std::shared_ptr, bool)>>(std::__function::__policy_storage const*, String const&, DB::DictionaryStructure const&, Poco::Util::AbstractConfiguration const&, String const&, std::shared_ptr&&, std::shared_ptr&&, bool) @ 0x000000000f8e33e5 in /usr/bin/clickhouse
21. DB::DictionaryFactory::create(String const&, Poco::Util::AbstractConfiguration const&, String const&, std::shared_ptr, bool) const @ 0x0000000010d3b87d in /usr/bin/clickhouse
22. DB::ExternalDictionariesLoader::create(String const&, Poco::Util::AbstractConfiguration const&, String const&, String const&) const @ 0x0000000011cb561c in /usr/bin/clickhouse
23. DB::ExternalLoader::LoadingDispatcher::doLoading(String const&, unsigned long, bool, unsigned long, bool, std::shared_ptr) @ 0x0000000011cc0897 in /usr/bin/clickhouse

24. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl), DB::ExternalLoader::LoadingDispatcher*, String&, unsigned long&, bool&, unsigned long&, bool, std::shared_ptr>(void (DB::ExternalLoader::LoadingDispatcher::*&&)(String const&, unsigned long, bool, unsigned long, bool, std::shared_ptr), DB::ExternalLoader::LoadingDispatcher*&&, String&, unsigned long&, bool&, unsigned long&, bool&&, std::shared_ptr&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000011cc595f in /usr/bin/clickhouse
25. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c826567 in /usr/bin/clickhouse
26. start_thread @ 0x0000000000007ea7 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
27. ? @ 0x00000000000fba2f in /usr/lib/x86_64-linux-gnu/libc-2.31.so
```

Query:
```
select dictGetFloat32('dict.chrt_price', 'price', 1)
```

All queries which contain this dictionary failed with this exception (other queries - ok).
Also this happened only at specific node (cluster with 5 nodes).
The problem was fixed by drop and create dictionary on the problematic node. Dictionary source= Clickhouse, layout=HASHED()

Upgrade steps:
```
1. sudo apt-get install clickhouse-server=23.9.5.29 clickhouse-client=23.9.5.29 clickhouse-common-static=23.9.5.29
2. service clickhouse-server restart
```

Any idea why it happened and how to avoid this problem at the future updates?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.