Configuration Improvements
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
h3. Known Configuration Issues
# *Key Generator Conflicts*
Changing the {{hoodie.datasource.write.keygenerator.class}} after initial table creation can lead to runtime exceptions. For instance, switching from {{SimpleKeyGenerator}} to {{GlobalDeleteKeyGenerator}} without recreating the table may raise a {{HoodieException}} due to mismatched metadata expectations.
🔗 https://medium.com/@life-is-short-so-enjoy-it/apache-hudi-exception-raised-when-using-different-keygenerator-d307d8efe7a1
# *Partition Path Field Data Type Conflicts*
If you specify partition field data types (e.g., {{{}inserted_at:TIMESTAMP{}}}) in one ingestion run and omit the type in another (e.g., just {{{}inserted_at{}}}), it can cause schema mismatch issues or ingestion failures.
🔗 [https://github.com/apache/hudi/issues/8372]
# *Flink SQL vs Hudi Config Overlap*
When using Flink SQL, specifying {{PRIMARY KEY}} and {{PARTITIONED BY}} can silently override {{hoodie.datasource.write.recordkey.field}} and {{{}hoodie.datasource.write.partitionpath.field{}}}, leading to confusing or unexpected ingestion behavior.
🔗 [https://github.com/apache/hudi/issues/12024]
# *Embedded Timeline Server with Flink*
Enabling the embedded timeline server ({{{}hoodie.embed.timeline.server=true{}}}) can lead to performance degradation or connectivity issues in environments like AWS Managed Flink, where cross-task communication is restricted. It is recommended to disable this in such cases.
🔗 [https://docs.aws.amazon.com/managed-flink/latest/java/troubleshooting-hudi.html]
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-9214
- Type: Improvement
- Fix version(s):
- 1.0.3
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the four configuration cases and the linked issues, especially HUDI-9214, issue 8372, and issue 12024. Trace the relevant Hudi and Flink configuration entry points and existing tests; the issue names no source files or acceptance criteria, so done requires confirming the intended behavior for each conflict and adding coverage for the agreed changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java, sql
- Domain
- cloud, data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100