apache / apache/hudi

[BUG] ALTER TABLE RENAME/DROP COLUMN fails with MissingSchemaFieldException under default configs since #13595

Open
#19,766 0 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

## Bug Description

**What happened:**

On a schema-on-read table with at least one commit, `ALTER TABLE ... RENAME COLUMN` and `ALTER TABLE ... DROP COLUMN` fail under the shipped defaults (`hoodie.avro.schema.validate=false`, `hoodie.datasource.write.schema.allow.auto.evolution.column.drop=false`):

```
org.apache.hudi.exception.MissingSchemaFieldException: Schema validation failed due to missing field. Fields missing from incoming schema: {price}
```

Both statements succeed once `hoodie.datasource.write.schema.allow.auto.evolution.column.drop=true` is set.

**To reproduce** (Spark SQL, COW or MOR):

```sql
set hoodie.schema.on.read.enable=true;
create table t (id int, name string, price double, ts long) using hudi
tblproperties (primaryKey = 'id', preCombineField = 'ts') location '/tmp/t';
insert into t values (1, 'a1', 10.0, 1000);
alter table t rename column price to newprice; -- MissingSchemaFieldException
alter table t drop column price; -- MissingSchemaFieldException
```

**Cause:**

#13595 (`0fe119a0cf1e`, 2025-07-25) added a bare `hoodieTable.validateSchema()` to `AlterTableCommand.commitWithSchema` (`AlterTableCommand.scala:280`) so that columns backing a secondary index cannot be evolved. `HoodieTable.validateSchema` only short-circuits when `!shouldValidate && allowProjection`; with `allowProjection=false` it reaches `HoodieSchemaCompatibility.checkSchemaCompatible`, whose first step is `findMissingFields(tableSchema, writerSchema)` and which throws for any table field absent from the new schema (`HoodieSchemaCompatibility.java:100-107`). A rename or a drop always produces exactly that.

The same commit added `allow.auto.evolution.column.drop=true` to the four pre-existing `TestSpark3DDL` rename/drop tests and to `TestTimeTravelTable`, which is why CI did not flag the change in behaviour. Every rename/drop test in the repo now carries that config; there is no test of the default path.

**Expected behavior:**

Schema-on-read rename/drop are explicit DDL, not an accidental writer-schema projection, so they should not depend on `allow.auto.evolution.column.drop`. The secondary-index guard from #13595 (`validateSecondaryIndexSchemaEvolution`) is the part of `validateSchema` that should keep running on this path; the missing-field check is the part that should not.

## Environment

Reproduced at `d291efccaad2`; the code path is unchanged on current master (`18ae8c349058`). Spark 3.5 / Scala 2.12 profile, local filesystem. Found while reviewing #19163, whose alter-table suite had to drop its rename/drop cases because of this.

## Logs and Stack Trace

```
org.apache.hudi.exception.MissingSchemaFieldException: Schema validation failed due to missing field. Fields missing from incoming schema: {price}
at org.apache.hudi.common.schema.HoodieSchemaCompatibility.checkSchemaCompatible(HoodieSchemaCompatibility.java:104)
at org.apache.hudi.table.HoodieTable.validateSchema(HoodieTable.java:936)
at org.apache.spark.sql.hudi.command.AlterTableCommand$.commitWithSchema(AlterTableCommand.scala:280)
at org.apache.spark.sql.hudi.command.AlterTableCommand.applyUpdateAction(AlterTableCommand.scala:180)
at org.apache.spark.sql.hudi.command.AlterTableCommand.run(AlterTableCommand.scala:60)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
```

The `DROP COLUMN` stack is identical apart from `applyDeleteAction` in place of `applyUpdateAction`.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at AlterTableCommand.scala:280 and trace commitWithSchema into HoodieTable.validateSchema and HoodieSchemaCompatibility.java:100-107. Run the affected rename/drop coverage in TestSpark3DDL and TestTimeTravelTable, checking the default configuration path. Done means schema-on-read RENAME COLUMN and DROP COLUMN work with shipped defaults while the secondary-index evolution guard remains effective.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, scala, spark, sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.