apache / apache/hudi

TestHoodieDeltaStreamerSchemaEvolutionQuick

Open
#16,409 1 comment 0 reactions 0 assignees View on GitHub
from-jira priority:high type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=22725&view=logs&j=dcedfe73-9485-5cc5-817a-73b61fc5dcb0&t=9df7def4-004b-5fb7-f042-da5d723783ad&s=859b8d9a-8fd6-5a5c-6f5e-f84f1990894e
{code:java}

[ERROR] Tests run: 29, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 514.307 s <<< FAILURE! - in org.apache.hudi.utilities.deltastreamer.TestHoodieDeltaStreamerSchemaEvolutionQuick
[ERROR] testReorderingColumn{String, Boolean, Boolean, Boolean}[1] Time elapsed: 13.21 s <<< ERROR!
org.apache.hudi.exception.SchemaCompatibilityException: Incoming batch schema is not compatible with the table's one
at org.apache.hudi.HoodieSchemaUtils$.deduceWriterSchema(HoodieSchemaUtils.scala:179)
at org.apache.hudi.HoodieSparkSqlWriter$.deduceWriterSchema(HoodieSparkSqlWriter.scala:147)
at org.apache.hudi.HoodieSparkSqlWriter.deduceWriterSchema(HoodieSparkSqlWriter.scala)
at org.apache.hudi.utilities.streamer.StreamSync.getDeducedSchemaProvider(StreamSync.java:671)
at org.apache.hudi.utilities.streamer.StreamSync.lambda$fetchNextBatchFromSource$5(StreamSync.java:612)
at org.apache.hudi.common.util.Option.map(Option.java:112)
at org.apache.hudi.utilities.streamer.StreamSync.fetchNextBatchFromSource(StreamSync.java:612)
at org.apache.hudi.utilities.streamer.StreamSync.fetchFromSourceAndPrepareRecords(StreamSync.java:524)
at org.apache.hudi.utilities.streamer.StreamSync.readFromSource(StreamSync.java:497)
at org.apache.hudi.utilities.streamer.StreamSync.syncOnce(StreamSync.java:400)
at org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.ingestOnce(HoodieStreamer.java:855)
at org.apache.hudi.utilities.ingestion.HoodieIngestionService.startIngestion(HoodieIngestionService.java:72)
at org.apache.hudi.common.util.Option.ifPresent(Option.java:101)
at org.apache.hudi.utilities.streamer.HoodieStreamer.sync(HoodieStreamer.java:211)
at org.apache.hudi.utilities.deltastreamer.TestHoodieDeltaStreamerSchemaEvolutionQuick.testReorderingColumn(TestHoodieDeltaStreamerSchemaEvolutionQuick.java:327) {code}

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-7468
- Type: Bug
- Epic: https://issues.apache.org/jira/browse/HUDI-4302

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with testReorderingColumn in TestHoodieDeltaStreamerSchemaEvolutionQuick at line 327, then trace the failure through StreamSync, HoodieSparkSqlWriter, and HoodieSchemaUtils. Run this targeted test and inspect the reported schema compatibility error. Done means the test's reordering scenario completes without the compatibility exception and the suite reports no errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, stream-processing, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.