[BUG] ALTER TABLE RENAME/DROP COLUMN fails with MissingSchemaFieldException under default configs since #13595
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
## Bug Description
**What happened:**
On a schema-on-read table with at least one commit, `ALTER TABLE ... RENAME COLUMN` and `ALTER TABLE ... DROP COLUMN` fail under the shipped defaults (`hoodie.avro.schema.validate=false`, `hoodie.datasource.write.schema.allow.auto.evolution.column.drop=false`):
```
org.apache.hudi.exception.MissingSchemaFieldException: Schema validation failed due to missing field. Fields missing from incoming schema: {price}
```
Both statements succeed once `hoodie.datasource.write.schema.allow.auto.evolution.column.drop=true` is set.
**To reproduce** (Spark SQL, COW or MOR):
```sql
set hoodie.schema.on.read.enable=true;
create table t (id int, name string, price double, ts long) using hudi
tblproperties (primaryKey = 'id', preCombineField = 'ts') location '/tmp/t';
insert into t values (1, 'a1', 10.0, 1000);
alter table t rename column price to newprice; -- MissingSchemaFieldException
alter table t drop column price; -- MissingSchemaFieldException
```
**Cause:**
#13595 (`0fe119a0cf1e`, 2025-07-25) added a bare `hoodieTable.validateSchema()` to `AlterTableCommand.commitWithSchema` (`AlterTableCommand.scala:280`) so that columns backing a secondary index cannot be evolved. `HoodieTable.validateSchema` only short-circuits when `!shouldValidate && allowProjection`; with `allowProjection=false` it reaches `HoodieSchemaCompatibility.checkSchemaCompatible`, whose first step is `findMissingFields(tableSchema, writerSchema)` and which throws for any table field absent from the new schema (`HoodieSchemaCompatibility.java:100-107`). A rename or a drop always produces exactly that.
The same commit added `allow.auto.evolution.column.drop=true` to the four pre-existing `TestSpark3DDL` rename/drop tests and to `TestTimeTravelTable`, which is why CI did not flag the change in behaviour. Every rename/drop test in the repo now carries that config; there is no test of the default path.
**Expected behavior:**
Schema-on-read rename/drop are explicit DDL, not an accidental writer-schema projection, so they should not depend on `allow.auto.evolution.column.drop`. The secondary-index guard from #13595 (`validateSecondaryIndexSchemaEvolution`) is the part of `validateSchema` that should keep running on this path; the missing-field check is the part that should not.
## Environment
Reproduced at `d291efccaad2`; the code path is unchanged on current master (`18ae8c349058`). Spark 3.5 / Scala 2.12 profile, local filesystem. Found while reviewing #19163, whose alter-table suite had to drop its rename/drop cases because of this.
## Logs and Stack Trace
```
org.apache.hudi.exception.MissingSchemaFieldException: Schema validation failed due to missing field. Fields missing from incoming schema: {price}
at org.apache.hudi.common.schema.HoodieSchemaCompatibility.checkSchemaCompatible(HoodieSchemaCompatibility.java:104)
at org.apache.hudi.table.HoodieTable.validateSchema(HoodieTable.java:936)
at org.apache.spark.sql.hudi.command.AlterTableCommand$.commitWithSchema(AlterTableCommand.scala:280)
at org.apache.spark.sql.hudi.command.AlterTableCommand.applyUpdateAction(AlterTableCommand.scala:180)
at org.apache.spark.sql.hudi.command.AlterTableCommand.run(AlterTableCommand.scala:60)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
```
The `DROP COLUMN` stack is identical apart from `applyDeleteAction` in place of `applyUpdateAction`.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at AlterTableCommand.scala:280 and trace commitWithSchema into HoodieTable.validateSchema and HoodieSchemaCompatibility.java:100-107. Run the affected rename/drop coverage in TestSpark3DDL and TestTimeTravelTable, checking the default configuration path. Done means schema-on-read RENAME COLUMN and DROP COLUMN work with shipped defaults while the secondary-index evolution guard remains effective.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, scala, spark, sql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 70/100