apache / apache/hudi

Setting partitioned by (partition_path) with nonpartitioned keygenerator in spark-sql will cause the colum to be null

Open
#15,589 0 comments 0 reactions 0 assignees View on GitHub
area:sql from-jira priority:high type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

When creating the table, for example: 
{code:java}
create table hudi_cow_pt_tbl (
id bigint,
name string,
ts bigint,
dt string,
hh string
) using hudi
tblproperties (
type = 'cow',
primaryKey = 'id',
preCombineField = 'ts'
 hoodie.table.keygenerator.class = 'org.apache.hudi.keygen.NonpartitionedKeyGenerator'
)
partitioned by (dt) {code}
When attempting to cache the dataframe I read, I got the error 

 
{code:java}
assertion failed: Empty partition column value in 'partition_path='
java.lang.AssertionError: assertion failed: Empty partition column value in 'partition_path=' {code}
 

This will cause dt to always be null when you read the record. I don't know if the data is stored as null or just reads as null. If this is due to implementation issues and the only fix would be to fail the table creation, I think that is preferable to the current behavior.  

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-5263
- Type: Bug

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the provided Spark SQL table definition and dataframe cache/read sequence, checking whether the partition_path value is written or only reconstructed as null. Trace the interaction between the partitioned-by declaration and NonpartitionedKeyGenerator. Done means the invalid combination is rejected clearly or dt is preserved correctly when records are read.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark, sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.