TestHoodieDeltaStreamerSchemaEvolutionQuick
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=22725&view=logs&j=dcedfe73-9485-5cc5-817a-73b61fc5dcb0&t=9df7def4-004b-5fb7-f042-da5d723783ad&s=859b8d9a-8fd6-5a5c-6f5e-f84f1990894e
{code:java}
[ERROR] Tests run: 29, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 514.307 s <<< FAILURE! - in org.apache.hudi.utilities.deltastreamer.TestHoodieDeltaStreamerSchemaEvolutionQuick
[ERROR] testReorderingColumn{String, Boolean, Boolean, Boolean}[1] Time elapsed: 13.21 s <<< ERROR!
org.apache.hudi.exception.SchemaCompatibilityException: Incoming batch schema is not compatible with the table's one
at org.apache.hudi.HoodieSchemaUtils$.deduceWriterSchema(HoodieSchemaUtils.scala:179)
at org.apache.hudi.HoodieSparkSqlWriter$.deduceWriterSchema(HoodieSparkSqlWriter.scala:147)
at org.apache.hudi.HoodieSparkSqlWriter.deduceWriterSchema(HoodieSparkSqlWriter.scala)
at org.apache.hudi.utilities.streamer.StreamSync.getDeducedSchemaProvider(StreamSync.java:671)
at org.apache.hudi.utilities.streamer.StreamSync.lambda$fetchNextBatchFromSource$5(StreamSync.java:612)
at org.apache.hudi.common.util.Option.map(Option.java:112)
at org.apache.hudi.utilities.streamer.StreamSync.fetchNextBatchFromSource(StreamSync.java:612)
at org.apache.hudi.utilities.streamer.StreamSync.fetchFromSourceAndPrepareRecords(StreamSync.java:524)
at org.apache.hudi.utilities.streamer.StreamSync.readFromSource(StreamSync.java:497)
at org.apache.hudi.utilities.streamer.StreamSync.syncOnce(StreamSync.java:400)
at org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.ingestOnce(HoodieStreamer.java:855)
at org.apache.hudi.utilities.ingestion.HoodieIngestionService.startIngestion(HoodieIngestionService.java:72)
at org.apache.hudi.common.util.Option.ifPresent(Option.java:101)
at org.apache.hudi.utilities.streamer.HoodieStreamer.sync(HoodieStreamer.java:211)
at org.apache.hudi.utilities.deltastreamer.TestHoodieDeltaStreamerSchemaEvolutionQuick.testReorderingColumn(TestHoodieDeltaStreamerSchemaEvolutionQuick.java:327) {code}
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-7468
- Type: Bug
- Epic: https://issues.apache.org/jira/browse/HUDI-4302
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with testReorderingColumn in TestHoodieDeltaStreamerSchemaEvolutionQuick at line 327, then trace the failure through StreamSync, HoodieSparkSqlWriter, and HoodieSchemaUtils. Run this targeted test and inspect the reported schema compatibility error. Done means the test's reordering scenario completes without the compatibility exception and the suite reports no errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, stream-processing, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100