Fail to add default partition
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
I got below error message:
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed in executing SQL ALTER TABLE `ods_xxx`.`xx` ADD IF NOT EXISTS PARTITION (`xx`='__HIVE_DEFAULT_PARTITION__') LOCATION 'xxxx/__HIVE_DEFAULT_PARTITION__'
at org.apache.hudi.hive.ddl.JDBCExecutor.runSQL(JDBCExecutor.java:70)
at org.apache.hudi.hive.ddl.QueryBasedDDLExecutor.lambda$addPartitionsToTable$0(QueryBasedDDLExecutor.java:124)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
at org.apache.hudi.hive.ddl.QueryBasedDDLExecutor.addPartitionsToTable(QueryBasedDDLExecutor.java:124)
at org.apache.hudi.hive.HoodieHiveSyncClient.addPartitionsToTable(HoodieHiveSyncClient.java:109)
at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:445)
at org.apache.hudi.hive.HiveSyncTool.syncAllPartitions(HiveSyncTool.java:399)
... 69 more
Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException [Error 10111]: Partition value contains a reserved substring (User value: __HIVE_DEFAULT_PARTITION__ Reserved substring: __HIVE_DEFAULT_PARTITION__)
Environment Description
Hudi version : 0.13.1
Spark version : 3.0.1
Hive version : 3.1
Hadoop version : 3.2.2
Storage (HDFS/S3/GCS..) :
Running on Docker? no :
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with HiveSyncTool.syncAllPartitions and syncPartitions, then trace HoodieHiveSyncClient.addPartitionsToTable into QueryBasedDDLExecutor.addPartitionsToTable and JDBCExecutor.runSQL. Reproduce the reported Hudi 0.13.1 failure with Hive 3.1 and determine how default partition values should be handled so synchronization no longer triggers Hive's reserved-substring error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100