Create "Type" configs for current configs that take in classes
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Take for example `hoodie.datasource.write.keygenerator.type` and its counterpart `hoodie.datasource.write.keygenerator.class`. We should make configs like keygenerator.type for configs currently like keygenerator.class. Take for example `hoodie.datasource.write.record.merger.impls`. If you want HoodieAvroRecordMerger you need to set the value to org.apache.hudi.common.model.HoodieAvroRecordMerger. However, if you want HoodieSparkRecordMerger, the full class path is org.apache.hudi.HoodieSparkRecordMerger. Differences like this are an easy cause for mistakes.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5761
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-5738
- Fix version(s):
- 1.1.0
---
## Comments
03/Apr/23 13:35;zxcoccer;Hi, I am trying to complete this issue, but there is some confusion in the process of completing the issue, how to get the `*_type` value elegantly, the method I can think of now is to correspond to a method for each `_type` Value, like this
{code:java}
switch (keyGeneratorTypeEnum){
case SIMPLE: return new SimpleAvroKeyGenerator(props);
case COMPLEX: return new ComplexAvroKeyGenerator(props);
case TIMESTAMP: return new TimestampBasedAvroKeyGenerator(props);
case NON_PARTITION: return new NonpartitionedAvroKeyGenerator(props);
case GLOBAL_DELETE: return new GlobalAvroDeleteKeyGenerator(props);
}
{code}
By the way, can you assign this ticket to me? I will try my best to complete it;;;
---
03/Apr/23 15:39;jonvex;Ok so keygenerator.type is now not such a good example because https://issues.apache.org/jira/browse/HUDI-5929 . So I will give a different example. In HoodieBootstrapConfig.java we have config "hoodie.bootstrap.mode.selector". We have selectors that are already created:
org.apache.hudi.client.bootstrap.selector.BootstrapRegexModeSelector
org.apache.hudi.client.bootstrap.selector.FullRecordBootstrapModeSelector
org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector
But you need to read the code to figure that out and it is long to type out. We want to create a new enum
{code:java}
public enum BootstrapSelectorType {
REGEX,
FULL_RECORD,
METADATA_ONLY,
CUSTOM
}
{code}
And create a new config "hoodie.bootstrap.mode.type" that takes in those config values
Then, in HoodieWriteConfig.java there is a method
{code:java}
public String getBootstrapModeSelectorClass() {
return getString(HoodieBootstrapConfig.MODE_SELECTOR_CLASS_NAME);
} {code}
We want to replace it with something like
{code:java}
public String getBootstrapModeSelectorClass() {
switch (getString(HoodieBootstrapConfig.MODE_SELECTOR_TYPE_NAME)) {
case REGEX: return "org.apache.hudi.client.bootstrap.selector.BootstrapRegexModeSelector";
case FULL_RECORD: return "org.apache.hudi.client.bootstrap.selector.FullRecordBootstrapModeSelector";
case METADATA_ONLY: return "
org.apache.hudi.client.bootstrap.selector.FullRecordBootstrapModeSelector";
case CUSTOM: return getString(HoodieBootstrapConfig.MODE_SELECTOR_CLASS_NAME);
}
}
{code}
Then repeat this with all other configs that take in class:
hoodie.datasource.write.payload.class
hoodie.datasource.write.record.merger.impls
hoodie.writestatus.class
hoodie.fileid.prefix.provider.class
for example. There are probably more than that.
To get assigned tickets, you need to email [dev@hudi.apache.org|mailto:dev@hudi.apache.org] and ask to be added as a contributor.
;;;
---
05/Apr/23 11:38;zxcoccer;Thank you very much for your reply. I will complete this task soon.:);;;
---
24/Apr/23 18:27;jonvex;Taking this over because there is a lot of documentation required;;;
---
24/Apr/23 18:40;jonvex;Additional classes to look at:
h3. -hoodie.datasource.hive_sync.partition_extractor_class-
h3. -hoodie.meta.sync.client.tool.class-
h3. hoodie.precommit.validators
h3. hoodie.filesystem.operation.retry.exceptions
h3. hoodie.bulkinsert.user.defined.partitioner.class
h3. hoodie.write.concurrency.early.conflict.detection.strategy
h3. hoodie.compaction.strategy
h3. hoodie.clustering.execution.strategy.class
h3. hoodie.clustering.plan.strategy.class
h3. hoodie.clustering.updates.strategy
h3. hoodie.bootstrap.full.input.provider
h3. hoodie.bootstrap.index.class
h3. hoodie.bootstrap.partitionpath.translator.class
h4. hoodie.write.commit.callback.class
h4. hoodie.write.lock.conflict.resolution.strategy
h4. hoodie.write.lock.provider
h4. hoodie.index.hbase.qps.allocator.class
h3. hoodie.meta.sync.datahub.dataset.identifier.class
h3. hoodie.schemaprovider.class
h3. hoodie.filesystem.view.remote.retry.exceptions[|https://hudi.apache.org/docs/configurations#hoodiefilesystemviewremoteretryexceptions]
;;;
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.