[SUPPORT] Hive sync failing with "Invalid default for field operationType" while migrating form Hudi 0.8 to 0.12
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
Upgrade from hudi 0.8 to hudi 0.12
Upgrade steps:
Use the hudi 0.12 program to write to the table created by the existing hudi 0.8, and use automatic upgrade
While migrating from Hudi 0.8 to Hudi 0.12 during HIVE_SYNC we are seeing "Avro default-value's validation exception for field: operationType". Please let us know if we are missing some step or configuration.
**Expected behavior**
Migration should happen without error.
**Environment Description**
* Hudi version : 0.12
* Spark version : 3.1
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```
Caused by: org.apache.hudi.exception.HoodieException: Got runtime exception when hive syncing
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:145)
at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:56)
... 60 more
Caused by: org.apache.avro.AvroTypeException:
Invalid default for field operationType: "" not a ["null",{"type":"string","avro.java.string":"String"}]
at org.apache.avro.Schema.validateDefault(Schema.java:1581)
at org.apache.avro.Schema.access$500(Schema.java:92)
at org.apache.avro.Schema$Field.(Schema.java:552)
at org.apache.avro.Schema.parse(Schema.java:1688)
at org.apache.avro.Schema.parse(Schema.java:1765)
at org.apache.avro.Schema.parse(Schema.java:1678)
at org.apache.avro.Schema$Parser.parse(Schema.java:1433)
at org.apache.avro.Schema$Parser.parse(Schema.java:1421)
at org.apache.hudi.common.table.log.block.HoodieAvroDataBlock$RecordIterator.getInstance(HoodieAvroDataBlock.java:180)
at org.apache.hudi.common.table.log.block.HoodieAvroDataBlock.deserializeRecords(HoodieAvroDataBlock.java:141)
at org.apache.hudi.common.table.log.block.HoodieDataBlock.readRecordsFromBlockPayload(HoodieDataBlock.java:189)
at org.apache.hudi.common.table.log.block.HoodieDataBlock.getRecordIterator(HoodieDataBlock.java:147)
at org.apache.hudi.common.table.timeline.HoodieArchivedTimeline.loadInstants(HoodieArchivedTimeline.java:260)
at org.apache.hudi.common.table.timeline.HoodieArchivedTimeline.(HoodieArchivedTimeline.java:109)
at org.apache.hudi.common.table.HoodieTableMetaClient.getArchivedTimeline(HoodieTableMetaClient.java:392)
at org.apache.hudi.sync.common.HoodieSyncClient.getWrittenPartitionsSince(HoodieSyncClient.java:120)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:226)
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:154)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:142)
... 61 more
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the HIVE_SYNC entry point in org.apache.hudi.hive.HiveSyncTool and trace the stack through HoodieSyncClient, HoodieArchivedTimeline, and HoodieAvroDataBlock to the Avro Schema.parse failure. Reproduce the Hudi 0.8-to-0.12 migration with Spark 3.1 and the supplied environment, then verify that Hive sync completes without the operationType default-value exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100