apache / apache/hudi

[SUPPORT] how to upgrade hudi table 2 to 5? The hudi files are not compatible with bucketIdFromFileId error

Open
#8,986 6 comments 0 reactions 0 assignees View on GitHub
area:upgrade priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**_Tips before filing an issue_**

**Describe the problem you faced**
The old table upgrade to 0.13.1 is failed with compatibility issues.
Do upgrade cmd can not work.

The reason is that the old data table file id is hexadecimal, and the new version is eight digits in decimal, which is incompatible and causes data writing exceptions

Old table files:
hudi/tables/test/dt=202204/**575acca4**-0e1d-402d-9f89-eacf2b869596-0_7-1541614-44548700_20220505124850.parquet

New table files:
hudi/tables/test/dt=202306/**00000019**-4b5b-46da-8c75-1f20ac2b9036-0_8-210-1959_20230615092038852.parquet

A clear and concise description of the problem.

**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment Description**

* Hudi version :0.13.1

* Spark version :3.2.1

* Hive version :3.1.2

* Hadoop version :3.2.2

* Storage (HDFS/S3/GCS..) :s3

* Running on Docker? (yes/no) :no

**Additional context**

Add any other context about the problem here.

**Stacktrace**

```Add the stacktrace of the error.```
Caused by: java.lang.NumberFormatException: For input string: "27b27208"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.parseInt(Integer.java:615)
at org.apache.hudi.index.bucket.BucketIdentifier.bucketIdFromFileId(BucketIdentifier.java:79)
at org.apache.hudi.index.bucket.HoodieSimpleBucketIndex.lambda$loadPartitionBucketIdFileIdMapping$0(HoodieSimpleBucketIndex.java:59)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.apache.hudi.index.bucket.HoodieSimpleBucketIndex.loadPartitionBucketIdFileIdMapping(HoodieSimpleBucketIndex.java:55)
at org.apache.hudi.index.bucket.HoodieSimpleBucketIndex.access$000(HoodieSimpleBucketIndex.java:39)
at org.apache.hudi.index.bucket.HoodieSimpleBucketIndex$SimpleBucketIndexLocationMapper.lambda$new$1(HoodieSimpleBucketIndex.java:89)
at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1321)
at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
at java.util.Iterator.forEachRemaining(Iterator.java:116)
at scala.collection.convert.Wrappers$IteratorWrapper.forEachRemaining(Wrappers.scala:31)
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
at org.apache.hudi.index.bucket.HoodieSimpleBucketIndex$SimpleBucketIndexLocationMapper.(HoodieSimpleBucketIndex.java:89)
at org.apache.hudi.index.bucket.HoodieSimpleBucketIndex.getLocationMapper(HoodieSimpleBucketIndex.java:78)
at org.apache.hudi.index.bucket.HoodieBucketIndex.tagLocation(HoodieBucketIndex.java:75)
at org.apache.hudi.table.action.commit.HoodieWriteHelper.tag(HoodieWriteHelper.java:55)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with org.apache.hudi.index.bucket.BucketIdentifier.java:79 and the mapping flow in HoodieSimpleBucketIndex.java:55-89. Reproduce the 0.13.1 upgrade or subsequent write using the hexadecimal file ID shown in the report, then determine the expected handling for both old and new file-name formats. Done means the upgrade/write path no longer fails with NumberFormatException and compatibility is covered by an appropriate test.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.