NPE in AvroSchemaEvolutionUtils.reconcileSchema when the incoming schema is null and the table schema has no fields
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Problem
`AvroSchemaEvolutionUtils.reconcileSchema` starts with
/* If incoming schema is null, we fall back on table schema. */
if (incomingSchema.isSchemaNull()) {
which handles an Avro NULL-typed schema but dereferences a Java `null`. A null incoming schema only reaches it because `reconcileSchemaRequirements` returns the *source* schema untouched whenever the *target* (table) schema is NULL-typed or has no fields, so a null source leaks through instead of being replaced by the table schema. The schema-on-read branch of `HoodieSchemaUtils.deduceWriterSchema` has the same gap (`sourceSchema.getFields` on the raw incoming schema).
### How it surfaces
A HoodieStreamer row source whose `SchemaProvider` returns `null` from `getTargetSchema()` (schema is derived from each batch), writing to a table whose latest commit carries a zero-column record schema (what a sync over column-less input leaves behind), fails every empty batch with
java.lang.NullPointerException
at org.apache.hudi.internal.schema.utils.AvroSchemaEvolutionUtils.reconcileSchema
at org.apache.hudi.HoodieSchemaUtils$.deduceWriterSchemaWithoutReconcile
at org.apache.hudi.HoodieSchemaUtils$.deduceWriterSchema
at org.apache.hudi.utilities.streamer.StreamSync.getDeducedSchemaProvider
at org.apache.hudi.utilities.streamer.StreamSync.fetchNextBatchFromSource
Because the failed sync never advances the checkpoint, the stream cannot recover on its own; a restart resumes at the same checkpoint and throws again. With a table schema that has columns the bug is masked, which is why it only shows on brand-new tables whose first ingested files carried no records.
### Expected
An absent incoming schema falls back to the table schema, as the comment promises, and an empty batch lands an empty commit that advances the checkpoint.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.