apache / apache/hudi

NPE in AvroSchemaEvolutionUtils.reconcileSchema when the incoming schema is null and the table schema has no fields

Open
#19,870 0 comments 0 reactions 1 assignee Claimed by @yihua View on GitHub
area:schema priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Problem

`AvroSchemaEvolutionUtils.reconcileSchema` starts with

/* If incoming schema is null, we fall back on table schema. */
if (incomingSchema.isSchemaNull()) {

which handles an Avro NULL-typed schema but dereferences a Java `null`. A null incoming schema only reaches it because `reconcileSchemaRequirements` returns the *source* schema untouched whenever the *target* (table) schema is NULL-typed or has no fields, so a null source leaks through instead of being replaced by the table schema. The schema-on-read branch of `HoodieSchemaUtils.deduceWriterSchema` has the same gap (`sourceSchema.getFields` on the raw incoming schema).

### How it surfaces

A HoodieStreamer row source whose `SchemaProvider` returns `null` from `getTargetSchema()` (schema is derived from each batch), writing to a table whose latest commit carries a zero-column record schema (what a sync over column-less input leaves behind), fails every empty batch with

java.lang.NullPointerException
at org.apache.hudi.internal.schema.utils.AvroSchemaEvolutionUtils.reconcileSchema
at org.apache.hudi.HoodieSchemaUtils$.deduceWriterSchemaWithoutReconcile
at org.apache.hudi.HoodieSchemaUtils$.deduceWriterSchema
at org.apache.hudi.utilities.streamer.StreamSync.getDeducedSchemaProvider
at org.apache.hudi.utilities.streamer.StreamSync.fetchNextBatchFromSource

Because the failed sync never advances the checkpoint, the stream cannot recover on its own; a restart resumes at the same checkpoint and throws again. With a table schema that has columns the bug is masked, which is why it only shows on brand-new tables whose first ingested files carried no records.

### Expected

An absent incoming schema falls back to the table schema, as the comment promises, and an empty batch lands an empty commit that advances the checkpoint.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.