[BUG] ALTER TABLE SET TBLPROPERTIES only updates the Spark catalog, never hoodie.properties, so no table config is changeable in-band
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
## Bug Description
**What happened:**
`ALTER TABLE ... SET TBLPROPERTIES` on a Hudi table only updates the Spark catalog entry and never rewrites `.hoodie/hoodie.properties`, so no Hudi table config is changeable in-band through SQL. `AlterTableCommand#applyPropertySet` builds a new `CatalogTable` and calls `catalog.alterTable(newTable)` -- nothing touches the table's own config file:
```scala
// AlterTableCommand.scala:202-212
def applyPropertySet(sparkSession: SparkSession): Unit = {
val catalog = sparkSession.sessionState.catalog
val properties = changes.map(_.asInstanceOf[SetProperty]).map(f => f.property -> f.value).toMap
val newTable = table.copy(
properties = table.properties ++ properties, ...)
catalog.alterTable(newTable)
}
```
The result is a silent divergence: the catalog says one thing, `hoodie.properties` (which the write and read paths actually consult) says another. The user believes the ALTER took effect, and nothing warns them otherwise.
**Motivating case:**
#19648 added a write-time rejection whose remedy would naturally be "turn the config off" --
but there is no in-band way to do that:
1. `ALTER TABLE t SET TBLPROPERTIES('hoodie.datasource.write.slash.separated.date.partitioning'='false')` succeeds and does nothing effective.
2. Passing the config as a write option instead trips `HoodieWriterUtils#validateTableConfig`'s config-diff rejection, since the option conflicts with the persisted table config.
So for any table config, the only escapes are `SaveMode.Overwrite` (recreates the table) or hand-editing `hoodie.properties`. The error message in #19648 now says "recreate the table" because that is the only honest advice.
**What you expected:**
Either `SET TBLPROPERTIES` on a `hoodie.`-prefixed key updates `hoodie.properties` (with validation of which configs are safe to change post-creation, since some are layout-affecting and genuinely immutable), or the command rejects `hoodie.`-prefixed keys outright instead of silently absorbing them into the catalog.
**Steps to reproduce:**
1. Create any Hudi table via Spark SQL.
2. `ALTER TABLE t SET TBLPROPERTIES('hoodie.datasource.write.slash.separated.date.partitioning'='true')`.
3. Inspect `.hoodie/hoodie.properties` -- the key is absent; inserts do not slash-separate.
**Suggested fix:**
Split table configs into mutable and immutable sets. For mutable ones, have `applyPropertySet`/`applyPropertyUnset` write through to `hoodie.properties` (via `HoodieTableConfig.update`); for immutable ones, throw with a message naming the recreate path. Silently updating only the catalog is the worst of both worlds.
## Environment
**Hudi version:** master (post-#19648) **Spark version:** 3.5 / 4.x (engine-agnostic: the gap is in hudi-spark-common)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with AlterTableCommand.scala, especially applyPropertySet and applyPropertyUnset, and trace how HoodieTableConfig.update could write changes to hoodie.properties. Review HoodieWriterUtils#validateTableConfig and the behavior described in the reproduction steps. Done means mutable hoodie.-prefixed settings are persisted and immutable settings are rejected with a recreate-table message, without catalog/file divergence.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100