apache / apache/hudi

Decouple virtual key with writing bloom filters to parquet files

Open
#15,606 1 comment 0 reactions 1 assignee Claimed by @yihua View on GitHub
area:index area:writer from-jira priority:critical status:pr-available type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 4h
Merged PRs (30d)
112

Description

When the virtual key feature is enabled by setting hoodie.populate.meta.fields to false, the bloom filters are not written to parquet base files in the write transactions.  Relevant logic in HoodieFileWriterFactory class:
{code:java}
private static HoodieFileWriter newParquetFileWriter(
String instantTime, Path path, HoodieWriteConfig config, Schema schema, HoodieTable hoodieTable,
TaskContextSupplier taskContextSupplier, boolean populateMetaFields) throws IOException {
return newParquetFileWriter(instantTime, path, config, schema, hoodieTable.getHadoopConf(),
taskContextSupplier, populateMetaFields, populateMetaFields);
}

private static HoodieFileWriter newParquetFileWriter(
String instantTime, Path path, HoodieWriteConfig config, Schema schema, Configuration conf,
TaskContextSupplier taskContextSupplier, boolean populateMetaFields, boolean enableBloomFilter) throws IOException {
Option filter = enableBloomFilter ? Option.of(createBloomFilter(config)) : Option.empty();
HoodieAvroWriteSupport writeSupport = new HoodieAvroWriteSupport(new AvroSchemaConverter(conf).convert(schema), schema, filter);

HoodieParquetConfig parquetConfig = new HoodieParquetConfig<>(writeSupport, config.getParquetCompressionCodec(),
config.getParquetBlockSize(), config.getParquetPageSize(), config.getParquetMaxFileSize(),
conf, config.getParquetCompressionRatio(), config.parquetDictionaryEnabled());

return new HoodieAvroParquetWriter<>(path, parquetConfig, instantTime, taskContextSupplier, populateMetaFields);
} {code}
Given that bloom filters are absent, when using Bloom Index on the same table, the writer encounters NPE (HUDI-5319).

We should decouple the virtual key feature with bloom filter and always write the bloom filters to the parquet files. 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-5323
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-5425

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.