NVIDIA / NVIDIA/cudf

[PERF]: Coalesce byte ranges before IO submission

Open
#23,538 1 comment 0 reactions 0 assignees View on GitHub
cudf-polars Performance
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

Merge adjacent parquet column chunk byte ranges before issuing pread or fadvise calls. We should investigate what the gaps between byte ranges (for filter and payload) are in our datasets to see how much benefit we can potentially get from this. This is already done by cucasCade and needs to be done by kvikio.

Note: Before we go implementing something, we really figure out how important this is for our benchmarks.

Contributor guide

Open the contributing guide

Research direction

Start by measuring the gaps between filter and payload byte ranges in representative datasets and assessing the impact on benchmarks, as requested in the issue. Compare the existing cucasCade behavior with kvikio's pread and fadvise submission path. Done means the benchmark evidence establishes whether this matters and, if worthwhile, adjacent ranges are coalesced before IO submission.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.