ClickHouse / ClickHouse/ClickHouse
Error Log Lacks Materialized View Identification via UUID
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
(you don't have to strictly follow this form)
**Describe the issue**
The error log does not contain the name of the failing materialized view.
**How to reproduce**
We have a table with ENGINE=Kafka and materialized views persisting data from Kafka to MergeTree tables. There can be several materialized views created from a Kafka table by different ClickHouse users.
If any user makes a mistake in their materialized view:
* ClickHouse stops consuming data from Kafka for all tables, which is not ideal as it halts the entire pipeline for all users.
* It is impossible to identify which materialized view is failing. The error log displays the following message without specifying
the failing materialized view:
```
2023.10.03 06:52:20.196005 [ 3249078 ] {} StorageKafka (tracking_ssp_auction_kafka): void DB::StorageKafka::threadFunc(size_t) Code: 60. DB::Exception: Table test.auction_by_zone2 does not exist. Maybe you
meant auction_by_zone?. (UNKNOWN_TABLE), Stack trace (when copying this message, always include the lines below):
0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) in /usr/bin/clickhouse
1. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) i
n /usr/bin/clickhouse
2. DB::IDatabase::getTable(String const&, std::shared_ptr) const in /usr/bin/clickhouse
3. DB::DatabaseCatalog::getTableImpl(DB::StorageID const&, std::shared_ptr, std::optional*) const in /usr/bin/clickhouse
4. DB::DatabaseCatalog::getTable(DB::StorageID const&, std::shared_ptr) const in /usr/bin/clickhouse
5. DB::StorageMaterializedView::getTargetTable() const in /usr/bin/clickhouse
6. DB::buildPushingToViewsChain(std::shared_ptr const&, std::shared_ptr const&, std::shared_ptr, std::shared_ptr const&, bool, std::shar
ed_ptr, std::shared_ptr, std::atomic*, bool, DB::Block const&) in /usr/bin/clickhouse
7. DB::InterpreterInsertQuery::buildSink(std::shared_ptr const&, std::shared_ptr const&, std::shared_ptr, std::shared_ptr,
std::atomic*) in /usr/bin/clickhouse
8. DB::InterpreterInsertQuery::buildChain(std::shared_ptr const&, std::shared_ptr const&, std::vector> const&, std::shared_ptr, std::atomic*) in /usr/bin/clickhouse
9. DB::buildPushingToViewsChain(std::shared_ptr const&, std::shared_ptr const&, std::shared_ptr, std::shared_ptr const&, bool, std::shar
ed_ptr, std::shared_ptr, std::atomic*, bool, DB::Block const&) in /usr/bin/clickhouse
10. DB::InterpreterInsertQuery::buildSink(std::shared_ptr const&, std::shared_ptr const&, std::shared_ptr, std::shared_ptr
, std::atomic*) in /usr/bin/clickhouse
11. DB::InterpreterInsertQuery::execute() in /usr/bin/clickhouse
12. DB::StorageKafka::threadFunc(unsigned long) in /usr/bin/clickhouse
13. DB::BackgroundSchedulePool::threadFunction() in /usr/bin/clickhouse
14. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl, StrongTypedef, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, St
rongTypedef, StrongTypedef, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) in /usr/bin/clic
khouse
15. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Pr
iority, std::optional, bool)::'lambda0'()>>(void*) in /usr/bin/clickhouse
16. ? in ?
17. ? in ?
(version 23.8.2.7 (official build))
```
For reference, the failing materialized view has the following CREATE statement:
```
CREATE MATERIALIZED VIEW test.auction_by_zone_mv TO test.auction_by_zone2 (
`h` DateTime, `cnt` UInt64, `dsp` String, `stage` String, `webId` UInt32, `zoneId` Int32
) AS
SELECT
h, count() AS cnt, dsp, stage, webId, zoneId
FROM (
SELECT
h, data.dsp, data.stage,
if(webzone.webId > 0, data.zoneId, 0) AS zoneId,
if(webzone.webId > 0, webzone.webId, 0) AS webId
FROM (
SELECT
toStartOfHour(ts) AS h,
tupleElement(imp, \'partner\') AS dsp,
tupleElement(imp, \'stage\') AS stage,
tupleElement(imp, \'zoneId\') AS zoneId,
tupleElement(imp, \'bidIds\') AS bidIds,
tupleElement(bidIds, \'cid\') AS cid
FROM kafka.tracking_ssp_auction
ARRAY JOIN impressions AS imp
WHERE h > \'2023-09-29 09:00:00\'
) AS data
LEFT JOIN test.stejskal_zones_webs AS webzone ON webzone.zoneId = data.zoneId
)
GROUP BY h, dsp, stage, webId, zoneId
```
* Which ClickHouse server version to use
23.8.2 revision 54465.
**Expected behavior**
The error log should ideally contain the UUID of the materialized table like so:
`2023.10.03 06:52:20.196005 [ 3249078 ] {UUID-HERE} `
This would allow users to identify the name of the materialized view using a SELECT statement from system.tables where uuid='...'. Additionally, it would be beneficial if system.errors also included a UUID column.
Contributor guide
Assessment
This issue has not been assessed yet.