[SUPPORT] Partition field value lost in table column
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
I am running spark-submit job to populate data into hudi tables from kafka topics. I have below properties set in my table-config.properties file:
hoodie.datasource.write.partitionpath.field=partitionFieldColumn
hoodie.datasource.hive_sync.table=tabledata
hoodie.datasource.hive_sync.partition_fields=partitionFieldColumn
hoodie.deltastreamer.keygen.timebased.timestamp.type=EPOCHMILLISECONDS
hoodie.deltastreamer.keygen.timebased.output.dateformat=yyyy/MM/dd
I am using "partitionFieldColumn" which is of datetime type. Using this, I want to have 3 level of partitioning (year -> month -> date). To avoid using time in the partitioning, "hoodie.deltastreamer.keygen.timebased.output.dateformat" property has the date format value. This results in correct partitioning levels but the "partitionFieldColumn" column created in the "tabledata" table also has the time field truncated, which is data loss.
Is there any way to retain the complete value of the "partitionFieldColumn" in the hive table without truncating the time field?
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the issue with the listed table-config.properties settings and a datetime partitionFieldColumn in the spark-submit ingestion flow. Compare the partition paths with the corresponding tabledata column value, then investigate how the timebased key generator and Hive sync handle the formatted partition field. Done means date-based partitions remain while the table column retains the complete datetime value.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- data-engineering, databases, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100