[SUPPORT] does hudi have plans to support default values
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Our dataset requires that no field is allowed to have null values, but hudi currently does not support default values, resulting in new fields having null values when querying historical data. The only way currently is to update historical data.
**Environment Description**
* Hudi version : 0.12.1
* Spark version : 3.1.3
* Hive version : 3.1.0
* Hadoop version : 3.1.1
* Storage (HDFS/S3/GCS..) : HDFS
* Running on Docker? (yes/no) : no
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing Hudi's schema-evolution and historical-query handling for the reported Hudi 0.12.1, Spark 3.1.3, Hive 3.1.0, Hadoop 3.1.1, and HDFS setup. The issue names no files or tests; done would require an agreed design for default values, behavior for existing records, and coverage of the no-null historical-data case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100