apache / apache/hudi

Improve how we determine spark partitions in bloom index based on record key size instead of key per bucket config

Open
#16,904 1 comment 0 reactions 1 assignee Assigned to @nsivabalan View on GitHub
area:index from-jira priority:high type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

As of now, the stage where we check the bloom filter and further lookup in the file for actual match, num spark partitions is determined based on {{hoodie.bloom.index.keys.per.bucket}} config. But in some cases, where the record key could be large (200 bytes), this might result in too much of data routed to each spark partition.
 
Ideally, we wanted to estimate the size of record keys and then determine the spark partitions based on that. say each record key is 50 bytes, we could route 2.5M keys to 1 spark task if we wanted to size it as 120Mb per spark task.
 
If the record key size is 200 bytes, that would reduce it to 625k entries to each spark task.
 
But getting hold of record key size may not be easy w/o collecting sample of records in the driver.
Alternatively, we can assume each column configured in record key field to be 36 bytes (UUID size). And depending on how many columns configured, we could derive some rough size w/o collecting the keys in the driver.
 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-9137
- Type: Improvement
- Fix version(s):
- 1.1.0

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.