ClickHouse / ClickHouse/ClickHouse

Error Log Lacks Materialized View Identification via UUID

Open
#55,214 0 comments 0 reactions 0 assignees View on GitHub
comp-materialized-view external usability
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

(you don't have to strictly follow this form)

**Describe the issue**
The error log does not contain the name of the failing materialized view.

**How to reproduce**

We have a table with ENGINE=Kafka and materialized views persisting data from Kafka to MergeTree tables. There can be several materialized views created from a Kafka table by different ClickHouse users.

If any user makes a mistake in their materialized view:

* ClickHouse stops consuming data from Kafka for all tables, which is not ideal as it halts the entire pipeline for all users.
* It is impossible to identify which materialized view is failing. The error log displays the following message without specifying

the failing materialized view:

```
2023.10.03 06:52:20.196005 [ 3249078 ] {} StorageKafka (tracking_ssp_auction_kafka): void DB::StorageKafka::threadFunc(size_t) Code: 60. DB::Exception: Table test.auction_by_zone2 does not exist. Maybe you
meant auction_by_zone?. (UNKNOWN_TABLE), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) in /usr/bin/clickhouse
1. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) i
n /usr/bin/clickhouse
2. DB::IDatabase::getTable(String const&, std::shared_ptr) const in /usr/bin/clickhouse
3. DB::DatabaseCatalog::getTableImpl(DB::StorageID const&, std::shared_ptr, std::optional*) const in /usr/bin/clickhouse
4. DB::DatabaseCatalog::getTable(DB::StorageID const&, std::shared_ptr) const in /usr/bin/clickhouse
5. DB::StorageMaterializedView::getTargetTable() const in /usr/bin/clickhouse
6. DB::buildPushingToViewsChain(std::shared_ptr const&, std::shared_ptr const&, std::shared_ptr, std::shared_ptr const&, bool, std::shar
ed_ptr, std::shared_ptr, std::atomic*, bool, DB::Block const&) in /usr/bin/clickhouse
7. DB::InterpreterInsertQuery::buildSink(std::shared_ptr const&, std::shared_ptr const&, std::shared_ptr, std::shared_ptr,
std::atomic*) in /usr/bin/clickhouse
8. DB::InterpreterInsertQuery::buildChain(std::shared_ptr const&, std::shared_ptr const&, std::vector> const&, std::shared_ptr, std::atomic*) in /usr/bin/clickhouse
9. DB::buildPushingToViewsChain(std::shared_ptr const&, std::shared_ptr const&, std::shared_ptr, std::shared_ptr const&, bool, std::shar
ed_ptr, std::shared_ptr, std::atomic*, bool, DB::Block const&) in /usr/bin/clickhouse
10. DB::InterpreterInsertQuery::buildSink(std::shared_ptr const&, std::shared_ptr const&, std::shared_ptr, std::shared_ptr
, std::atomic*) in /usr/bin/clickhouse
11. DB::InterpreterInsertQuery::execute() in /usr/bin/clickhouse
12. DB::StorageKafka::threadFunc(unsigned long) in /usr/bin/clickhouse
13. DB::BackgroundSchedulePool::threadFunction() in /usr/bin/clickhouse
14. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl, StrongTypedef, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, St
rongTypedef, StrongTypedef, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) in /usr/bin/clic
khouse
15. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Pr
iority, std::optional, bool)::'lambda0'()>>(void*) in /usr/bin/clickhouse
16. ? in ?
17. ? in ?
(version 23.8.2.7 (official build))
```

For reference, the failing materialized view has the following CREATE statement:

```
CREATE MATERIALIZED VIEW test.auction_by_zone_mv TO test.auction_by_zone2 (
`h` DateTime, `cnt` UInt64, `dsp` String, `stage` String, `webId` UInt32, `zoneId` Int32
) AS
SELECT
h, count() AS cnt, dsp, stage, webId, zoneId
FROM (
SELECT
h, data.dsp, data.stage,
if(webzone.webId > 0, data.zoneId, 0) AS zoneId,
if(webzone.webId > 0, webzone.webId, 0) AS webId
FROM (
SELECT
toStartOfHour(ts) AS h,
tupleElement(imp, \'partner\') AS dsp,
tupleElement(imp, \'stage\') AS stage,
tupleElement(imp, \'zoneId\') AS zoneId,
tupleElement(imp, \'bidIds\') AS bidIds,
tupleElement(bidIds, \'cid\') AS cid
FROM kafka.tracking_ssp_auction
ARRAY JOIN impressions AS imp
WHERE h > \'2023-09-29 09:00:00\'
) AS data
LEFT JOIN test.stejskal_zones_webs AS webzone ON webzone.zoneId = data.zoneId
)
GROUP BY h, dsp, stage, webId, zoneId
```

* Which ClickHouse server version to use
23.8.2 revision 54465.

**Expected behavior**
The error log should ideally contain the UUID of the materialized table like so:

`2023.10.03 06:52:20.196005 [ 3249078 ] {UUID-HERE} `

This would allow users to identify the name of the materialized view using a SELECT statement from system.tables where uuid='...'. Additionally, it would be beneficial if system.errors also included a UUID column.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.