apache / apache/hudi

Create "Type" configs for current configs that take in classes

Open
#15,780 1 comment 0 reactions 1 assignee Claimed by @yihua View on GitHub
area:config from-jira priority:high status:pr-available type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Take for example `hoodie.datasource.write.keygenerator.type` and its counterpart `hoodie.datasource.write.keygenerator.class`. We should make configs like keygenerator.type for configs currently like keygenerator.class. Take for example `hoodie.datasource.write.record.merger.impls`. If you want HoodieAvroRecordMerger you need to set the value to org.apache.hudi.common.model.HoodieAvroRecordMerger. However, if you want HoodieSparkRecordMerger, the full class path is org.apache.hudi.HoodieSparkRecordMerger. Differences like this are an easy cause for mistakes.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-5761
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-5738
- Fix version(s):
- 1.1.0

---

## Comments

03/Apr/23 13:35;zxcoccer;Hi, I am trying to complete this issue, but there is some confusion in the process of completing the issue, how to get the `*_type` value elegantly, the method I can think of now is to correspond to a method for each `_type` Value, like this
{code:java}
switch (keyGeneratorTypeEnum){      
case SIMPLE:         return new SimpleAvroKeyGenerator(props);      
case COMPLEX:         return new ComplexAvroKeyGenerator(props);      
case TIMESTAMP:       return new TimestampBasedAvroKeyGenerator(props);      
case NON_PARTITION:   return new NonpartitionedAvroKeyGenerator(props);  
case GLOBAL_DELETE:   return new GlobalAvroDeleteKeyGenerator(props);  
}
{code}
By the way, can you assign this ticket to me? I will try my best to complete it;;;

---

03/Apr/23 15:39;jonvex;Ok so keygenerator.type is now not such a good example because https://issues.apache.org/jira/browse/HUDI-5929 . So I will give a different example. In HoodieBootstrapConfig.java we have config "hoodie.bootstrap.mode.selector". We have selectors that are already created:

org.apache.hudi.client.bootstrap.selector.BootstrapRegexModeSelector

org.apache.hudi.client.bootstrap.selector.FullRecordBootstrapModeSelector

org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector

But you need to read the code to figure that out and it is long to type out. We want to create a new enum 

 
{code:java}
public enum BootstrapSelectorType {
REGEX,
FULL_RECORD,
METADATA_ONLY,
CUSTOM
}
{code}
And create a new config "hoodie.bootstrap.mode.type" that takes in those config values

 

Then, in HoodieWriteConfig.java there is a method 

 
{code:java}
public String getBootstrapModeSelectorClass() {
return getString(HoodieBootstrapConfig.MODE_SELECTOR_CLASS_NAME);
} {code}
We want to replace it with something like 

 
{code:java}
public String getBootstrapModeSelectorClass() {
switch (getString(HoodieBootstrapConfig.MODE_SELECTOR_TYPE_NAME)) {
case REGEX: return "org.apache.hudi.client.bootstrap.selector.BootstrapRegexModeSelector";
case FULL_RECORD: return "org.apache.hudi.client.bootstrap.selector.FullRecordBootstrapModeSelector";
case METADATA_ONLY: return "
org.apache.hudi.client.bootstrap.selector.FullRecordBootstrapModeSelector";
case CUSTOM: return getString(HoodieBootstrapConfig.MODE_SELECTOR_CLASS_NAME);
}
}
{code}
Then repeat this with all other configs that take in class:

hoodie.datasource.write.payload.class

hoodie.datasource.write.record.merger.impls

hoodie.writestatus.class

hoodie.fileid.prefix.provider.class

for example. There are probably more than that.

 

 

 

 

To get assigned tickets, you need to email [dev@hudi.apache.org|mailto:dev@hudi.apache.org] and ask to be added as a contributor.

 ;;;

---

05/Apr/23 11:38;zxcoccer;Thank you very much for your reply. I will complete this task soon.:);;;

---

24/Apr/23 18:27;jonvex;Taking this over because there is a lot of documentation required;;;

---

24/Apr/23 18:40;jonvex;Additional classes to look at:
h3. -hoodie.datasource.hive_sync.partition_extractor_class-
h3. -hoodie.meta.sync.client.tool.class-
h3. hoodie.precommit.validators
h3. hoodie.filesystem.operation.retry.exceptions

 
h3. hoodie.bulkinsert.user.defined.partitioner.class

 
h3. hoodie.write.concurrency.early.conflict.detection.strategy

 
h3. hoodie.compaction.strategy

 
h3. hoodie.clustering.execution.strategy.class

 
h3. hoodie.clustering.plan.strategy.class
h3. hoodie.clustering.updates.strategy

 
h3. hoodie.bootstrap.full.input.provider

 
h3. hoodie.bootstrap.index.class

 
h3. hoodie.bootstrap.partitionpath.translator.class

 
h4. hoodie.write.commit.callback.class

 
h4. hoodie.write.lock.conflict.resolution.strategy

 
h4. hoodie.write.lock.provider

 
h4. hoodie.index.hbase.qps.allocator.class

 
h3. hoodie.meta.sync.datahub.dataset.identifier.class

 
h3. hoodie.schemaprovider.class

 
h3. hoodie.filesystem.view.remote.retry.exceptions[​|https://hudi.apache.org/docs/configurations#hoodiefilesystemviewremoteretryexceptions]

 ;;;

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.