apache / apache/hudi

[SUPPORT] META_SYNC_DECODE_PARTITION=true is not fully supported when using HIVE_SYNC_MODE=hms

Open
#13,516 0 comments 0 reactions 0 assignees View on GitHub
component:catalog-sync engine:spark type:community-support
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

When using Hive sync in HMS mode (`HIVE_SYNC_MODE = hms`) and setting `URL_ENCODE_PARTITIONING` to `true` and `META_SYNC_DECODE_PARTITION` to `true` it **does not** decode the partition key.

**To Reproduce**

1. Set `hoodie.datasource.hive_sync.mode=hms`
2. Set `hoodie.datasource.write.partitionpath.urlencode=true`
3. Set `hoodie.meta.sync.decode_partition=true`
4. Write data with partition key (e.g. `partition=2025-06-10 05:00:00`)
5. Inspect the database

**Expected behavior**

Partition key should be URL-decoded (e.g. `partition=2025-06-10 05:00:00`)

**Actual behavior**

Partition key is URL-encoded (e.g. `partition=2025-06-10%2005%3A00%3A00`)

**Environment Description**

* Hudi version : 0.15.0

* Spark version : 3.5.4

* Hive version : 2.3.9-amzn-4

* Hadoop version : 3.4.1

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) : No

**Additional context**

There seems to be a bug in [HMSDDLExecutor.java](https://github.com/apache/hudi/blob/master/hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java) where the `META_SYNC_DECODE_PARTITION` option **is not** respected and the partition key is not decoded when calling [addPartitionsToTable()](https://github.com/apache/hudi/blob/master/hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java#L191).

In contrast, the `META_SYNC_DECODE_PARTITION` option **is** respected when calling [dropPartitionsToTable()](https://github.com/apache/hudi/blob/master/hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java#L250) as it calls [HivePartitionUtil.getPartitionClauseForDrop()](https://github.com/apache/hudi/blob/master/hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/HivePartitionUtil.java#L57).

Therefore this seems like a bug as there is an inconsistency in how partition keys are handled when adding and dropping them.

**Note:** When setting `HIVE_SYNC_MODE = hiveql` or `HIVE_SYNC_MODE = jdbc` the `META_SYNC_DECODE_PARTITION` option is respected during both adding and dropping the partition key so this bug is limited to the `hms` option only.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java at addPartitionsToTable(), then compare it with dropPartitionsToTable() and HivePartitionUtil.getPartitionClauseForDrop(). Reproduce with HMS mode, URL-encoded partitioning, and META_SYNC_DECODE_PARTITION enabled; done means the HMS database shows the decoded partition key for both adding and dropping partitions.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java, spark
Domain
data-engineering, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.