apache / apache/hudi

Improve the generation of metadata records of bloom filters

Open
#15,703 1 comment 0 reactions 0 assignees View on GitHub
area:metadata-table from-jira priority:critical type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

When updating the metadata table's bloom_filter partition, we read the parquet footers to get the bloom filters before converting them to metadata table records for upsert.  This part can be affected by throttling, so we'll need to generate the bloom filters whenever parquet files are written and flow them into the metadata writer.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-5587
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-1292

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing updates to the metadata table’s bloom_filter partition and the parquet-write path, the relevant entry points named in the issue. Examine how bloom filters currently move from parquet footers into metadata writer records, then identify tests covering throttling and metadata upserts. Done means bloom filters are generated during parquet writes and reach the metadata writer without relying on footer reads during the update.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.