apache / apache/hudi

[SUPPORT] Append Mode should support close the bloom filter option

Open
#8,274 14 comments 0 reactions 0 assignees View on GitHub
engine:flink priority:medium type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Write in insert mode, but also write bloomfilter according to recordkey at the same time, I think you can set an option to turn off this function to increase write throughput

I did not find the corresponding setting in the 0.13 branch, it should be enabled by default

```
private static HoodieRowDataFileWriter newParquetInternalRowFileWriter(
Path path, HoodieWriteConfig writeConfig, RowType rowType, HoodieTable table)
throws IOException {
BloomFilter filter = BloomFilterFactory.createBloomFilter(
writeConfig.getBloomFilterNumEntries(),
writeConfig.getBloomFilterFPP(),
writeConfig.getDynamicBloomFilterMaxNumEntries(),
writeConfig.getBloomFilterType());
HoodieRowDataParquetWriteSupport writeSupport =
new HoodieRowDataParquetWriteSupport(table.getHadoopConf(), rowType, filter);
return new HoodieRowDataParquetWriter(
path, new HoodieParquetConfig<>(
writeSupport,
writeConfig.getParquetCompressionCodec(),
writeConfig.getParquetBlockSize(),
writeConfig.getParquetPageSize(),
writeConfig.getParquetMaxFileSize(),
writeSupport.getHadoopConf(),
writeConfig.getParquetCompressionRatio(),
writeConfig.parquetDictionaryEnabled()));
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at newParquetInternalRowFileWriter and inspect how the BloomFilter is created and passed to HoodieRowDataParquetWriteSupport in the 0.13 branch. Trace the related write configuration options and define completion as an append-mode option that can disable Bloom filter writing while preserving the default behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.