apache / apache/hudi

Config Simplification

Open
#15,775 0 comments 0 reactions 1 assignee Claimed by @yihua View on GitHub
area:config from-jira priority:high type:epic
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

EPIC for simplifying configs and improving usability around configs in Hudi

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-5738
- Type: Epic
- Fix version(s):
- 1.1.0
- Attachment(s):
- 29/Feb/24 08:31;geserdugarov;Hudi configuration parameters.xlsx;https://issues.apache.org/jira/secure/attachment/13067149/Hudi+configuration+parameters.xlsx

---

## Comments

29/Feb/24 08:25;geserdugarov;[~guoyihua], [~jonvex], hi!

I'm trying to figure out how to handle huge amount of configuration parameters in Hudi. So, I started from data collecting and want to share current results.
Note, that all numbers are for the master branch, commit d0929341d0ad25b3ce587d7f229d2fcdf7796768 at Feb 26, 2024.

The number of parameters is really huge, current total number is {*}899 parameters{*}.

By modules:
* hudi-flink-datasource - 130 parameters,
* hudi-spark-datasource - 45,
* hudi-client-common - 294,
* hudi-common - 184,
* hudi-utilities - 143,
* kafka, hive, sync, aws, gcp, and datahub related - 103.

740 parameters marked as advanced (only 9 from them are marked as deprecated).
For now, we have 159 not advanced parameters.;;;

---

29/Feb/24 08:31;geserdugarov; [^Hudi configuration parameters.xlsx]

Here my table with all configurations. You can look at all of them, sort, filter, analyze default values.;;;

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.