Performance improvements for streaming DAG write with secondary index
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Couple of performance improvements on HUDI-9340.
1. While fetching secondary key from file group, we can project the secondary key itself instead of reading the entire record.
2. In HoodieAppendHandle, we can avoid reading the file slice twice to compute the secondary index changes. We can use the new records available in the handle and merge with previous file slice to compute the secondary index related changes.
3. We currently use toString to get the string representation of secondary key. We need to ensure this works with all data types - like date, timestamp.
[https://github.com/apache/hudi/blob/e017d85d76b5a2332e96ce0b7e4b2a552f98dadc/hudi-common/src/main/java/org/apache/hudi/metadata/SecondaryIndexRecordGenerationUtils.java#L259]
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-9546
- Type: Sub-task
- Parent: https://issues.apache.org/jira/browse/HUDI-9616
- Fix version(s):
- 1.1.0
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.