apache / apache/hudi

[SUPPORT] Hive sync failing with "Invalid default for field operationType" while migrating form Hudi 0.8 to 0.12

Open
#8,210 6 comments 0 reactions 0 assignees View on GitHub
area:aws area:schema issue:version-compatibility priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**_Tips before filing an issue_**

- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?

- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.

**Describe the problem you faced**
Upgrade from hudi 0.8 to hudi 0.12

Upgrade steps:
Use the hudi 0.12 program to write to the table created by the existing hudi 0.8, and use automatic upgrade

While migrating from Hudi 0.8 to Hudi 0.12 during HIVE_SYNC we are seeing "Avro default-value's validation exception for field: operationType". Please let us know if we are missing some step or configuration.

**Expected behavior**
Migration should happen without error.

**Environment Description**

* Hudi version : 0.12

* Spark version : 3.1

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) : no

**Additional context**

Add any other context about the problem here.

**Stacktrace**

```

Caused by: org.apache.hudi.exception.HoodieException: Got runtime exception when hive syncing
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:145)
at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:56)
... 60 more
Caused by: org.apache.avro.AvroTypeException:
Invalid default for field operationType: "" not a ["null",{"type":"string","avro.java.string":"String"}]
at org.apache.avro.Schema.validateDefault(Schema.java:1581)
at org.apache.avro.Schema.access$500(Schema.java:92)
at org.apache.avro.Schema$Field.(Schema.java:552)
at org.apache.avro.Schema.parse(Schema.java:1688)
at org.apache.avro.Schema.parse(Schema.java:1765)
at org.apache.avro.Schema.parse(Schema.java:1678)
at org.apache.avro.Schema$Parser.parse(Schema.java:1433)
at org.apache.avro.Schema$Parser.parse(Schema.java:1421)
at org.apache.hudi.common.table.log.block.HoodieAvroDataBlock$RecordIterator.getInstance(HoodieAvroDataBlock.java:180)
at org.apache.hudi.common.table.log.block.HoodieAvroDataBlock.deserializeRecords(HoodieAvroDataBlock.java:141)
at org.apache.hudi.common.table.log.block.HoodieDataBlock.readRecordsFromBlockPayload(HoodieDataBlock.java:189)
at org.apache.hudi.common.table.log.block.HoodieDataBlock.getRecordIterator(HoodieDataBlock.java:147)
at org.apache.hudi.common.table.timeline.HoodieArchivedTimeline.loadInstants(HoodieArchivedTimeline.java:260)
at org.apache.hudi.common.table.timeline.HoodieArchivedTimeline.(HoodieArchivedTimeline.java:109)
at org.apache.hudi.common.table.HoodieTableMetaClient.getArchivedTimeline(HoodieTableMetaClient.java:392)
at org.apache.hudi.sync.common.HoodieSyncClient.getWrittenPartitionsSince(HoodieSyncClient.java:120)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:226)
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:154)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:142)
... 61 more
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the HIVE_SYNC entry point in org.apache.hudi.hive.HiveSyncTool and trace the stack through HoodieSyncClient, HoodieArchivedTimeline, and HoodieAvroDataBlock to the Avro Schema.parse failure. Reproduce the Hudi 0.8-to-0.12 migration with Spark 3.1 and the supplied environment, then verify that Hive sync completes without the operationType default-value exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.