apache / apache/hudi

Fix partitioning style when partition is inferred from partitionBy

Open
#16,413 2 comments 0 reactions 1 assignee Claimed by @nsivabalan View on GitHub
from-jira priority:blocker status:pr-available type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

When inferring partition from partitionBy() arguments and hive style partitioning is enabled, we observe that the partitioining style is not uniformed for multi-level partition. Directory structure is as follows:
partition=2015
                       |- 03
                             |- 15
                             |- 16

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-7484
- Type: Task
- Fix version(s):
- 1.1.0

---

## Comments

15/Mar/24 06:02;wombatukun;Hi! [~codope] Can you provide the full code sample?

It looks like it's not the case of bug in inferring partition from partitionBy(), just  the values of partition field are "2015/03/15", "2015/03/16". If data in partitioning column may contain special characters, then URL_ENCODE_PARTITIONING should be used while writing (which is false by default), and it results in directory structure like this:

partition=2015%2F03%2F15

partition=2015%2F03%2F16

 

Partitioning by multiple fields (trip_type, rider) without slashes in its values works correctly, producing directories like:

trip_type=BLACK

         |- rider=rider-000

trip_type=UBERX

         |- rider=rider-000

 ;;;

---

27/Mar/24 05:12;vinay.bhat;[~codope] Do you have some pointers or a test case? I want to understand what this is about and how to proceed.;;;

---

30/Sep/24 03:27;codope;This is not a bug per se, but bad user experience when partition contain slashes and users have enabled hive style partitioning.

I think if users have enabled hive style partitioning, then we should also enable url encoded partitioning. However, this is a breaking change and needs more thought.;;;

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.