[SUPPORT] META_SYNC_DECODE_PARTITION=true is not fully supported when using HIVE_SYNC_MODE=hms
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem you faced**
When using Hive sync in HMS mode (`HIVE_SYNC_MODE = hms`) and setting `URL_ENCODE_PARTITIONING` to `true` and `META_SYNC_DECODE_PARTITION` to `true` it **does not** decode the partition key.
**To Reproduce**
1. Set `hoodie.datasource.hive_sync.mode=hms`
2. Set `hoodie.datasource.write.partitionpath.urlencode=true`
3. Set `hoodie.meta.sync.decode_partition=true`
4. Write data with partition key (e.g. `partition=2025-06-10 05:00:00`)
5. Inspect the database
**Expected behavior**
Partition key should be URL-decoded (e.g. `partition=2025-06-10 05:00:00`)
**Actual behavior**
Partition key is URL-encoded (e.g. `partition=2025-06-10%2005%3A00%3A00`)
**Environment Description**
* Hudi version : 0.15.0
* Spark version : 3.5.4
* Hive version : 2.3.9-amzn-4
* Hadoop version : 3.4.1
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : No
**Additional context**
There seems to be a bug in [HMSDDLExecutor.java](https://github.com/apache/hudi/blob/master/hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java) where the `META_SYNC_DECODE_PARTITION` option **is not** respected and the partition key is not decoded when calling [addPartitionsToTable()](https://github.com/apache/hudi/blob/master/hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java#L191).
In contrast, the `META_SYNC_DECODE_PARTITION` option **is** respected when calling [dropPartitionsToTable()](https://github.com/apache/hudi/blob/master/hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java#L250) as it calls [HivePartitionUtil.getPartitionClauseForDrop()](https://github.com/apache/hudi/blob/master/hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/HivePartitionUtil.java#L57).
Therefore this seems like a bug as there is an inconsistency in how partition keys are handled when adding and dropping them.
**Note:** When setting `HIVE_SYNC_MODE = hiveql` or `HIVE_SYNC_MODE = jdbc` the `META_SYNC_DECODE_PARTITION` option is respected during both adding and dropping the partition key so this bug is limited to the `hms` option only.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java at addPartitionsToTable(), then compare it with dropPartitionsToTable() and HivePartitionUtil.getPartitionClauseForDrop(). Reproduce with HMS mode, URL-encoded partitioning, and META_SYNC_DECODE_PARTITION enabled; done means the HMS database shows the decoded partition key for both adding and dropping partitions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java, spark
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100