Improve the generation of metadata records of bloom filters
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
When updating the metadata table's bloom_filter partition, we read the parquet footers to get the bloom filters before converting them to metadata table records for upsert. This part can be affected by throttling, so we'll need to generate the bloom filters whenever parquet files are written and flow them into the metadata writer.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5587
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-1292
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing updates to the metadata table’s bloom_filter partition and the parquet-write path, the relevant entry points named in the issue. Examine how bloom filters currently move from parquet footers into metadata writer records, then identify tests covering throttling and metadata upserts. Done means bloom filters are generated during parquet writes and reach the metadata writer without relying on footer reads during the update.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100