apache / apache/gluten

[CH] Fix GlutenClickHouseMergeTreeWriteOnHDFSSuite and GlutenClickHouseMergeTreeWriteOnS3Suite after gluten domain name changes

Open
#5,229 0 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

CH (ClickHouse)

### Bug description

[2024-03-29T12:02:49.115Z] Caused by: org.apache.gluten.exception.GlutenException: Unknown storage policy `__s3_main`
[2024-03-29T12:02:49.115Z] 0. /home/jenkins/agent/workspace/gluten/gluten-ci/gluten/cpp-ch/build/../ClickHouse/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000b1546bb
[2024-03-29T12:02:49.115Z] 1. ../ClickHouse/contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception(int, FormatStringHelperImpl::type>, String&&) @ 0x000000000584c9a3
[2024-03-29T12:02:49.115Z] 2. /home/jenkins/agent/workspace/gluten/gluten-ci/gluten/cpp-ch/build/../ClickHouse/src/Disks/StoragePolicy.cpp:0: DB::StoragePolicySelector::get(String const&) const @ 0x000000000e2ecb2b
[2024-03-29T12:02:49.115Z] 3. /home/jenkins/agent/workspace/gluten/gluten-ci/gluten/cpp-ch/build/../ClickHouse/src/Interpreters/Context.cpp:0: DB::Context::getStoragePolicy(String const&) const @ 0x000000000e2bf582
[2024-03-29T12:02:49.115Z] 4. /home/jenkins/agent/workspace/gluten/gluten-ci/gluten/cpp-ch/build/../ClickHouse/src/Storages/MergeTree/MergeTreeData.cpp:0: DB::MergeTreeData::getStoragePolicy() const @ 0x000000000f2ffe8b
[2024-03-29T12:02:49.115Z] 5. /home/jenkins/agent/workspace/gluten/gluten-ci/gluten/cpp-ch/build/../local-engine/Storages/Mergetree/SparkMergeTreeWriter.cpp:183: local_engine::SparkMergeTreeWriter::writeTempPart(DB::BlockWithPartition&, std::shared_ptr const&) @ 0x000000000b51d4df
[2024-03-29T12:02:49.115Z] 6. /home/jenkins/agent/workspace/gluten/gluten-ci/gluten/cpp-ch/build/../local-engine/Storages/Mergetree/SparkMergeTreeWriter.cpp:79: local_engine::SparkMergeTreeWriter::finalize() @ 0x000000000b51c720
[2024-03-29T12:02:49.115Z] 7. /home/jenkins/agent/workspace/gluten/gluten-ci/gluten/cpp-ch/build/../local-engine/local_engine_jni.cpp:0: Java_org_apache_spark_sql_execution_datasources_CHDatasourceJniWrapper_closeMergeTreeWriter @ 0x0000000005bf235f
[2024-03-29T12:02:49.115Z]
[2024-03-29T12:02:49.115Z] at org.apache.spark.sql.execution.datasources.CHDatasourceJniWrapper.closeMergeTreeWriter(Native Method)
[2024-03-29T12:02:49.115Z] at org.apache.spark.sql.execution.datasources.v1.clickhouse.MergeTreeOutputWriter.close(MergeTreeOutputWriter.scala:50)
[2024-03-29T12:02:49.115Z] at org.apache.spark.sql.execution.datasources.v1.clickhouse.MergeTreeFileFormatDataWriter.releaseCurrentWriter(MergeTreeFileFormatDataWriter.scala:72)
[2024-03-29T12:02:49.115Z] at org.apache.spark.sql.execution.datasources.v1.clickhouse.MergeTreeBaseDynamicPartitionDataWriter.renewCurrentWriter(MergeTreeFileFormatDataWriter.scala:304)
[2024-03-29T12:02:49.115Z] at org.apache.spark.sql.execution.datasources.v1.clickhouse.MergeTreeDynamicPartitionDataSingleWriter.beforeWrite(MergeTreeFileFormatDataWriter.scala:429)
[2024-03-29T12:02:49.115Z] at org.apache.spark.sql.execution.datasources.v1.clickhouse.MergeTreeDynamicPartitionDataSingleWriter.write(MergeTreeFileFormatDataWriter.scala:449)
[2024-03-29T12:02:49.115Z] at org.apache.spark.sql.execution.datasources.v1.clickhouse.MergeTreeFileFormatDataWriter.writeWithMetrics(MergeTreeFileFormatDataWriter.scala:97)
[2024-03-29T12:02:49.115Z] at org.apache.spark.sql.execution.datasources.v1.clickhouse.MergeTreeFileFormatDataWriter.writeWithIterator(MergeTreeFileFormatDataWriter.scala:104)

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.