apache / apache/hudi

testUpgradeDowngrade failing intermittently

Open
#14,014 0 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Bug Description

**What happened:**

Currently the test method `testUpgradeDowngrade` in class `TestUpgradeDowngrade` fails on the CI for the following.

**What you expected:**

Test should not fail, so need to investigate to see what is causing this NPE.

**Steps to reproduce:**
Running the test locally usually passes so may be hard to repro.

### Environment

**Hudi version:**
master
**Query engine:** (Spark/Flink/Trino etc)
Spark
**Relevant configs:**
Run test locally

### Logs and Stack Trace

```
Error: testUpgradeOrDowngrade{HoodieTableVersion, HoodieTableVersion}[4] Time elapsed: 0.932 s <<< ERROR!
org.apache.hudi.exception.HoodieException: Failed to downgrade LSM timeline to old archived format
at org.apache.hudi.table.upgrade.EightToSevenDowngradeHandler.downgradeFromLSMTimeline(EightToSevenDowngradeHandler.java:233)
at org.apache.hudi.table.upgrade.EightToSevenDowngradeHandler.downgrade(EightToSevenDowngradeHandler.java:132)
at org.apache.hudi.table.upgrade.UpgradeDowngrade.downgrade(UpgradeDowngrade.java:328)
at org.apache.hudi.table.upgrade.UpgradeDowngrade.run(UpgradeDowngrade.java:230)
at org.apache.hudi.table.upgrade.TestUpgradeDowngrade.testUpgradeOrDowngrade(TestUpgradeDowngrade.java:112)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.util.Optional.ifPresent(Optional.java:159)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647)
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:272)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:272)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:272)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at java.util.ArrayList.forEach(ArrayList.java:1259)
Caused by: org.apache.hudi.exception.HoodieCommitException: Failed to archive commits
at org.apache.hudi.client.timeline.versioning.v1.TimelineArchiverV1.flushArchiveEntries(TimelineArchiverV1.java:173)
at org.apache.hudi.table.upgrade.EightToSevenDowngradeHandler$ArchiveEntryFlusher.close(EightToSevenDowngradeHandler.java:274)
at org.apache.hudi.table.upgrade.EightToSevenDowngradeHandler.downgradeFromLSMTimeline(EightToSevenDowngradeHandler.java:229)
... 45 more
Caused by: java.lang.NullPointerException
at org.apache.hudi.avro.HoodieAvroUtils.indexedRecordToBytesStream(HoodieAvroUtils.java:148)
at org.apache.hudi.avro.HoodieAvroUtils.avroToBytesStream(HoodieAvroUtils.java:144)
at org.apache.hudi.common.model.HoodieAvroIndexedRecord.getAvroBytes(HoodieAvroIndexedRecord.java:325)
at org.apache.hudi.common.table.log.block.HoodieAvroDataBlock.serializeRecords(HoodieAvroDataBlock.java:120)
at org.apache.hudi.common.table.log.block.HoodieDataBlock.getContentBytes(HoodieDataBlock.java:132)
at org.apache.hudi.common.table.log.HoodieLogFormatWriter.appendBlocks(HoodieLogFormatWriter.java:155)
at org.apache.hudi.common.table.log.HoodieLogFormatWriter.appendBlock(HoodieLogFormatWriter.java:132)
at org.apache.hudi.client.timeline.versioning.v1.TimelineArchiverV1.writeToFile(TimelineArchiverV1.java:447)
at org.apache.hudi.client.timeline.versioning.v1.TimelineArchiverV1.flushArchiveEntries(TimelineArchiverV1.java:171)
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with TestUpgradeDowngrade.testUpgradeOrDowngrade and follow the stack trace through EightToSevenDowngradeHandler, TimelineArchiverV1, and HoodieAvroUtils. Run the upgrade/downgrade test repeatedly under Spark and inspect the NPE path during archive serialization. Done means the test no longer fails intermittently in CI.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.