Fix partitioning style when partition is inferred from partitionBy
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
When inferring partition from partitionBy() arguments and hive style partitioning is enabled, we observe that the partitioining style is not uniformed for multi-level partition. Directory structure is as follows:
partition=2015
|- 03
|- 15
|- 16
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-7484
- Type: Task
- Fix version(s):
- 1.1.0
---
## Comments
15/Mar/24 06:02;wombatukun;Hi! [~codope] Can you provide the full code sample?
It looks like it's not the case of bug in inferring partition from partitionBy(), just the values of partition field are "2015/03/15", "2015/03/16". If data in partitioning column may contain special characters, then URL_ENCODE_PARTITIONING should be used while writing (which is false by default), and it results in directory structure like this:
partition=2015%2F03%2F15
partition=2015%2F03%2F16
Partitioning by multiple fields (trip_type, rider) without slashes in its values works correctly, producing directories like:
trip_type=BLACK
|- rider=rider-000
trip_type=UBERX
|- rider=rider-000
;;;
---
27/Mar/24 05:12;vinay.bhat;[~codope] Do you have some pointers or a test case? I want to understand what this is about and how to proceed.;;;
---
30/Sep/24 03:27;codope;This is not a bug per se, but bad user experience when partition contain slashes and users have enabled hive style partitioning.
I think if users have enabled hive style partitioning, then we should also enable url encoded partitioning. However, this is a breaking change and needs more thought.;;;
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.