testUpgradeDowngrade failing intermittently
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Bug Description
**What happened:**
Currently the test method `testUpgradeDowngrade` in class `TestUpgradeDowngrade` fails on the CI for the following.
**What you expected:**
Test should not fail, so need to investigate to see what is causing this NPE.
**Steps to reproduce:**
Running the test locally usually passes so may be hard to repro.
### Environment
**Hudi version:**
master
**Query engine:** (Spark/Flink/Trino etc)
Spark
**Relevant configs:**
Run test locally
### Logs and Stack Trace
```
Error: testUpgradeOrDowngrade{HoodieTableVersion, HoodieTableVersion}[4] Time elapsed: 0.932 s <<< ERROR!
org.apache.hudi.exception.HoodieException: Failed to downgrade LSM timeline to old archived format
at org.apache.hudi.table.upgrade.EightToSevenDowngradeHandler.downgradeFromLSMTimeline(EightToSevenDowngradeHandler.java:233)
at org.apache.hudi.table.upgrade.EightToSevenDowngradeHandler.downgrade(EightToSevenDowngradeHandler.java:132)
at org.apache.hudi.table.upgrade.UpgradeDowngrade.downgrade(UpgradeDowngrade.java:328)
at org.apache.hudi.table.upgrade.UpgradeDowngrade.run(UpgradeDowngrade.java:230)
at org.apache.hudi.table.upgrade.TestUpgradeDowngrade.testUpgradeOrDowngrade(TestUpgradeDowngrade.java:112)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.util.Optional.ifPresent(Optional.java:159)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647)
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:272)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:272)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:272)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at java.util.ArrayList.forEach(ArrayList.java:1259)
Caused by: org.apache.hudi.exception.HoodieCommitException: Failed to archive commits
at org.apache.hudi.client.timeline.versioning.v1.TimelineArchiverV1.flushArchiveEntries(TimelineArchiverV1.java:173)
at org.apache.hudi.table.upgrade.EightToSevenDowngradeHandler$ArchiveEntryFlusher.close(EightToSevenDowngradeHandler.java:274)
at org.apache.hudi.table.upgrade.EightToSevenDowngradeHandler.downgradeFromLSMTimeline(EightToSevenDowngradeHandler.java:229)
... 45 more
Caused by: java.lang.NullPointerException
at org.apache.hudi.avro.HoodieAvroUtils.indexedRecordToBytesStream(HoodieAvroUtils.java:148)
at org.apache.hudi.avro.HoodieAvroUtils.avroToBytesStream(HoodieAvroUtils.java:144)
at org.apache.hudi.common.model.HoodieAvroIndexedRecord.getAvroBytes(HoodieAvroIndexedRecord.java:325)
at org.apache.hudi.common.table.log.block.HoodieAvroDataBlock.serializeRecords(HoodieAvroDataBlock.java:120)
at org.apache.hudi.common.table.log.block.HoodieDataBlock.getContentBytes(HoodieDataBlock.java:132)
at org.apache.hudi.common.table.log.HoodieLogFormatWriter.appendBlocks(HoodieLogFormatWriter.java:155)
at org.apache.hudi.common.table.log.HoodieLogFormatWriter.appendBlock(HoodieLogFormatWriter.java:132)
at org.apache.hudi.client.timeline.versioning.v1.TimelineArchiverV1.writeToFile(TimelineArchiverV1.java:447)
at org.apache.hudi.client.timeline.versioning.v1.TimelineArchiverV1.flushArchiveEntries(TimelineArchiverV1.java:171)
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with TestUpgradeDowngrade.testUpgradeOrDowngrade and follow the stack trace through EightToSevenDowngradeHandler, TimelineArchiverV1, and HoodieAvroUtils. Run the upgrade/downgrade test repeatedly under Spark and inspect the NPE path during archive serialization. Done means the test no longer fails intermittently in CI.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100