apache / apache/hudi

Fail to add default partition

Open
#10,154 8 comments 0 reactions 0 assignees View on GitHub
area:sql priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

I got below error message:

Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed in executing SQL ALTER TABLE `ods_xxx`.`xx` ADD IF NOT EXISTS PARTITION (`xx`='__HIVE_DEFAULT_PARTITION__') LOCATION 'xxxx/__HIVE_DEFAULT_PARTITION__'
at org.apache.hudi.hive.ddl.JDBCExecutor.runSQL(JDBCExecutor.java:70)
at org.apache.hudi.hive.ddl.QueryBasedDDLExecutor.lambda$addPartitionsToTable$0(QueryBasedDDLExecutor.java:124)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
at org.apache.hudi.hive.ddl.QueryBasedDDLExecutor.addPartitionsToTable(QueryBasedDDLExecutor.java:124)
at org.apache.hudi.hive.HoodieHiveSyncClient.addPartitionsToTable(HoodieHiveSyncClient.java:109)
at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:445)
at org.apache.hudi.hive.HiveSyncTool.syncAllPartitions(HiveSyncTool.java:399)
... 69 more
Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException [Error 10111]: Partition value contains a reserved substring (User value: __HIVE_DEFAULT_PARTITION__ Reserved substring: __HIVE_DEFAULT_PARTITION__)

Environment Description

Hudi version : 0.13.1

Spark version : 3.0.1

Hive version : 3.1

Hadoop version : 3.2.2

Storage (HDFS/S3/GCS..) :

Running on Docker? no :

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with HiveSyncTool.syncAllPartitions and syncPartitions, then trace HoodieHiveSyncClient.addPartitionsToTable into QueryBasedDDLExecutor.addPartitionsToTable and JDBCExecutor.runSQL. Reproduce the reported Hudi 0.13.1 failure with Hive 3.1 and determine how default partition values should be handled so synchronization no longer triggers Hive's reserved-substring error.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.