[SUPPORT] Append Mode should support close the bloom filter option
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Write in insert mode, but also write bloomfilter according to recordkey at the same time, I think you can set an option to turn off this function to increase write throughput
I did not find the corresponding setting in the 0.13 branch, it should be enabled by default
```
private static HoodieRowDataFileWriter newParquetInternalRowFileWriter(
Path path, HoodieWriteConfig writeConfig, RowType rowType, HoodieTable table)
throws IOException {
BloomFilter filter = BloomFilterFactory.createBloomFilter(
writeConfig.getBloomFilterNumEntries(),
writeConfig.getBloomFilterFPP(),
writeConfig.getDynamicBloomFilterMaxNumEntries(),
writeConfig.getBloomFilterType());
HoodieRowDataParquetWriteSupport writeSupport =
new HoodieRowDataParquetWriteSupport(table.getHadoopConf(), rowType, filter);
return new HoodieRowDataParquetWriter(
path, new HoodieParquetConfig<>(
writeSupport,
writeConfig.getParquetCompressionCodec(),
writeConfig.getParquetBlockSize(),
writeConfig.getParquetPageSize(),
writeConfig.getParquetMaxFileSize(),
writeSupport.getHadoopConf(),
writeConfig.getParquetCompressionRatio(),
writeConfig.parquetDictionaryEnabled()));
}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at newParquetInternalRowFileWriter and inspect how the BloomFilter is created and passed to HoodieRowDataParquetWriteSupport in the 0.13 branch. Trace the related write configuration options and define completion as an append-mode option that can disable Bloom filter writing while preserving the default behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100