re-use spark config for parquet timestamp format instead of having our own config
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
We have two diff configs to set the right timestamp format.
"hoodie.parquet.outputtimestamptype": "TIMESTAMP_MICROS",
and spark config
--conf spark.sql.parquet.outputTimestampType=TIMESTAMP_MICROS
We should deprecate our own config and just rely on spark's configs.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-3555
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-1239
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the implementation of hoodie.parquet.outputtimestamptype and the handling of Spark's spark.sql.parquet.outputTimestampType setting. Check existing tests around Parquet timestamp writing before determining the deprecation behavior. Done means the project relies on Spark's configuration and the separate Hudi configuration is deprecated consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100