[PERF]: Coalesce byte ranges before IO submission
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
Merge adjacent parquet column chunk byte ranges before issuing pread or fadvise calls. We should investigate what the gaps between byte ranges (for filter and payload) are in our datasets to see how much benefit we can potentially get from this. This is already done by cucasCade and needs to be done by kvikio.
Note: Before we go implementing something, we really figure out how important this is for our benchmarks.
Contributor guide
Research direction
Start by measuring the gaps between filter and payload byte ranges in representative datasets and assessing the impact on benchmarks, as requested in the issue. Compare the existing cucasCade behavior with kvikio's pread and fadvise submission path. Done means the benchmark evidence establishes whether this matters and, if worthwhile, adjacent ranges are coalesced before IO submission.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100