apache / apache/hudi

[BUG] ALTER TABLE SET TBLPROPERTIES only updates the Spark catalog, never hoodie.properties, so no table config is changeable in-band

Open
#19,722 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

## Bug Description

**What happened:**

`ALTER TABLE ... SET TBLPROPERTIES` on a Hudi table only updates the Spark catalog entry and never rewrites `.hoodie/hoodie.properties`, so no Hudi table config is changeable in-band through SQL. `AlterTableCommand#applyPropertySet` builds a new `CatalogTable` and calls `catalog.alterTable(newTable)` -- nothing touches the table's own config file:

```scala
// AlterTableCommand.scala:202-212
def applyPropertySet(sparkSession: SparkSession): Unit = {
val catalog = sparkSession.sessionState.catalog
val properties = changes.map(_.asInstanceOf[SetProperty]).map(f => f.property -> f.value).toMap
val newTable = table.copy(
properties = table.properties ++ properties, ...)
catalog.alterTable(newTable)
}
```

The result is a silent divergence: the catalog says one thing, `hoodie.properties` (which the write and read paths actually consult) says another. The user believes the ALTER took effect, and nothing warns them otherwise.

**Motivating case:**

#19648 added a write-time rejection whose remedy would naturally be "turn the config off" --
but there is no in-band way to do that:

1. `ALTER TABLE t SET TBLPROPERTIES('hoodie.datasource.write.slash.separated.date.partitioning'='false')` succeeds and does nothing effective.
2. Passing the config as a write option instead trips `HoodieWriterUtils#validateTableConfig`'s config-diff rejection, since the option conflicts with the persisted table config.

So for any table config, the only escapes are `SaveMode.Overwrite` (recreates the table) or hand-editing `hoodie.properties`. The error message in #19648 now says "recreate the table" because that is the only honest advice.

**What you expected:**

Either `SET TBLPROPERTIES` on a `hoodie.`-prefixed key updates `hoodie.properties` (with validation of which configs are safe to change post-creation, since some are layout-affecting and genuinely immutable), or the command rejects `hoodie.`-prefixed keys outright instead of silently absorbing them into the catalog.

**Steps to reproduce:**

1. Create any Hudi table via Spark SQL.
2. `ALTER TABLE t SET TBLPROPERTIES('hoodie.datasource.write.slash.separated.date.partitioning'='true')`.
3. Inspect `.hoodie/hoodie.properties` -- the key is absent; inserts do not slash-separate.

**Suggested fix:**

Split table configs into mutable and immutable sets. For mutable ones, have `applyPropertySet`/`applyPropertyUnset` write through to `hoodie.properties` (via `HoodieTableConfig.update`); for immutable ones, throw with a message naming the recreate path. Silently updating only the catalog is the worst of both worlds.

## Environment

**Hudi version:** master (post-#19648) **Spark version:** 3.5 / 4.x (engine-agnostic: the gap is in hudi-spark-common)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with AlterTableCommand.scala, especially applyPropertySet and applyPropertyUnset, and trace how HoodieTableConfig.update could write changes to hoodie.properties. Review HoodieWriterUtils#validateTableConfig and the behavior described in the reproduction steps. Done means mutable hoodie.-prefixed settings are persisted and immutable settings are rejected with a recreate-table message, without catalog/file divergence.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.