Improve how we determine spark partitions in bloom index based on record key size instead of key per bucket config
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
As of now, the stage where we check the bloom filter and further lookup in the file for actual match, num spark partitions is determined based on {{hoodie.bloom.index.keys.per.bucket}} config. But in some cases, where the record key could be large (200 bytes), this might result in too much of data routed to each spark partition.
Ideally, we wanted to estimate the size of record keys and then determine the spark partitions based on that. say each record key is 50 bytes, we could route 2.5M keys to 1 spark task if we wanted to size it as 120Mb per spark task.
If the record key size is 200 bytes, that would reduce it to 625k entries to each spark task.
But getting hold of record key size may not be easy w/o collecting sample of records in the driver.
Alternatively, we can assume each column configured in record key field to be 36 bytes (UUID size). And depending on how many columns configured, we could derive some rough size w/o collecting the keys in the driver.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-9137
- Type: Improvement
- Fix version(s):
- 1.1.0
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.