apache / apache/hudi

Code refactoring for config defaults and wiring

Open
#15,829 0 comments 0 reactions 1 assignee Assigned to @yihua View on GitHub
area:config from-jira priority:high type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

See [https://github.com/apache/hudi/pull/7912/files] for more details

Note that this ticket may be split into separate improvement areas for clarify.

Below are tracked in separate tickets:

CLEANER_COMMITS_RETAINED: when either "hoodie.cleaner.commits.retained", "hoodie.cleaner.hours.retained", or "hoodie.cleaner.fileversions.retained" is set, should we automatically use the corresponding clean policy?

Clustering around group size: PLAN_STRATEGY_MAX_BYTES_PER_OUTPUT_FILEGROUP

LAYOUT_TYPE

ORDERING_FIELD (PAYLOAD_ORDERING_FIELD_PROP_KEY)

PAYLOAD_CLASS_NAME ("hoodie.compaction.payload.class")

KEYGENERATOR_TYPE (auto inference)

Low ROI which can be punt:

PLAN_STRATEGY_CLASS_NAME to enum

MERGE_ALLOW_DUPLICATE_ON_INSERTS_ENABLE

EQUALITY_SQL_QUERIES

These should be untouched:

EMBEDDED_TIMELINE_SERVER_REUSE_ENABLED (Flink may still need this)

DYNAMODB_ENDPOINT_URL (this is inDynamoDbBasedLockConfig and we only use the config key.  There is no ROI for adding infer logic.)

AWS_ACCESS_KEY, AWS_SECRET_KEY (These configs should only be necessary if no environmental variables are not set (to confirm with code); no action should be needed on changing the code)

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-5892
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-5738
- Fix version(s):
- 1.1.0

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.