ClickHouse / ClickHouse/ClickHouse
S3Queue & schema evolution problems
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
Imagine we have the following s3_queue table
```
CREATE TABLE s3_queue_engine_table
(
`name` String,
`value` UInt32
)
ENGINE = S3Queue('s3://test/*', NOSIGN, CSV)
SETTINGS mode = 'unordered', after_processing = 'keep', keeper_path = '/clickhouse/tables/s3_queue_engine_table'
```
Now we need to add new column to that, while preserving the progress (i.e. remember which files were already read and which not).
The easiest option is to recreate the table with a new schema, but:
```
drop table s3_queue_engine_table sync;
CREATE TABLE s3_queue_engine_table
(
`name` String,
`value` UInt32,
`new_column` UInt8
)
ENGINE = S3Queue('s3://test/*', NOSIGN, CSV)
SETTINGS mode = 'unordered', after_processing = 'keep', keeper_path = '/clickhouse/tables/s3_queue_engine_table'
Query id: 145efefa-9ab0-4324-a444-6f0a091a6c05
Elapsed: 0.008 sec.
Received exception from server (version 24.9.1):
Code: 122. DB::Exception: Received from localhost:9000. DB::Exception: Table columns structure in ZooKeeper is different from local table structure. Local columns:
columns format version: 1
3 columns:
`name` String
`value` UInt32
`new_column` UInt8
Zookeeper columns:
columns format version: 1
2 columns:
`name` String
`value` UInt32
. Stack trace:
0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x0000000014c1f892
1. ./build/./src/Common/Exception.cpp:109: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000bdb28b9
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x00000000069b5aec
3. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x00000000069b6ecb
4. ./build/./src/Storages/ObjectStorageQueue/ObjectStorageQueueMetadata.cpp:246: DB::ObjectStorageQueueMetadata::initialize(std::shared_ptr const&, DB::StorageInMemoryMetadata const&) @ 0x0000000011db7125
5. ./build/./src/Storages/ObjectStorageQueue/StorageObjectStorageQueue.cpp:184: DB::StorageObjectStorageQueue::StorageObjectStorageQueue(std::unique_ptr>, std::shared_ptr, DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr, std::optional, DB::ASTStorage*, DB::LoadingStrictnessLevel) @ 0x0000000011e23455
6. ./contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::StorageObjectStorageQueue* std::construct_at[abi:v15007]>, std::shared_ptr, DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr, std::optional&, DB::ASTStorage* const&, DB::LoadingStrictnessLevel const&, DB::StorageObjectStorageQueue*>(DB::StorageObjectStorageQueue*, std::unique_ptr>&&, std::shared_ptr&&, DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr&&, std::optional&, DB::ASTStorage* const&, DB::LoadingStrictnessLevel const&) @ 0x0000000011e204a6
7. ./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:298: std::shared_ptr std::allocate_shared[abi:v15007], std::unique_ptr>, std::shared_ptr, DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr, std::optional&, DB::ASTStorage* const&, DB::LoadingStrictnessLevel const&, void>(std::allocator const&, std::unique_ptr>&&, std::shared_ptr&&, DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::shared_ptr&&, std::optional&, DB::ASTStorage* const&, DB::LoadingStrictnessLevel const&) @ 0x0000000011e2029a
8. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr DB::createQueueStorage(DB::StorageFactory::Arguments const&) @ 0x0000000011e1eade
9. ./contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001149c3a5
10. ./build/./src/Interpreters/InterpreterCreateQuery.cpp:1723: DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr>&, DB::LoadingStrictnessLevel) @ 0x000000001079c2d1
11. ./build/./src/Interpreters/InterpreterCreateQuery.cpp:1547: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x00000000107972ea
12. ./build/./src/Interpreters/InterpreterCreateQuery.cpp:2047: DB::InterpreterCreateQuery::execute() @ 0x00000000107a15b1
13. ./build/./src/Interpreters/executeQuery.cpp:1238: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x0000000010b9c92c
14. ./build/./src/Interpreters/executeQuery.cpp:1404: DB::executeQuery(String const&, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x0000000010b990b5
15. ./build/./src/Server/TCPHandler.cpp:564: DB::TCPHandler::runImpl() @ 0x0000000011f91afa
16. ./build/./src/Server/TCPHandler.cpp:2484: DB::TCPHandler::run() @ 0x0000000011fa9499
17. ./build/./base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x0000000014cc4607
18. ./build/./base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x0000000014cc4ada
19. ./build/./base/poco/Foundation/src/ThreadPool.cpp:205: Poco::PooledThread::run() @ 0x0000000014c6fcf2
20. ./base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000014c6d7e3
21. ? @ 0x00007f114e094ac3
22. ? @ 0x00007f114e126850
. (INCOMPATIBLE_COLUMNS)
```
Why do we need to cross check the schema in the s3queue at all? That check (and storing the schema) seems redundant.
Contributor guide
Assessment
This issue has not been assessed yet.