Setting partitioned by (partition_path) with nonpartitioned keygenerator in spark-sql will cause the colum to be null
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
When creating the table, for example:
{code:java}
create table hudi_cow_pt_tbl (
id bigint,
name string,
ts bigint,
dt string,
hh string
) using hudi
tblproperties (
type = 'cow',
primaryKey = 'id',
preCombineField = 'ts'
hoodie.table.keygenerator.class = 'org.apache.hudi.keygen.NonpartitionedKeyGenerator'
)
partitioned by (dt) {code}
When attempting to cache the dataframe I read, I got the error
{code:java}
assertion failed: Empty partition column value in 'partition_path='
java.lang.AssertionError: assertion failed: Empty partition column value in 'partition_path=' {code}
This will cause dt to always be null when you read the record. I don't know if the data is stored as null or just reads as null. If this is due to implementation issues and the only fix would be to fail the table creation, I think that is preferable to the current behavior.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5263
- Type: Bug
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the provided Spark SQL table definition and dataframe cache/read sequence, checking whether the partition_path value is written or only reconstructed as null. Trace the interaction between the partitioned-by declaration and NonpartitionedKeyGenerator. Done means the invalid combination is rejected clearly or dt is preserved correctly when records are read.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark, sql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100