NVIDIA / NVIDIA/cudf

[PERF] Use per-file sizes for greedy partition binning

Open
#23,859 0 comments 0 reactions 1 assignee Claimed by @Matt711 View on GitHub
cudf-polars
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

`scan_partition_plan` computes a single uniform factor from the mean estimated decompressed size of the projected columns across sampled footers, then applies that factor to every file. When file sizes vary across a dataset, the mean is a poor representative. Small files that should be fused into a single task are split instead; large full-sized files and small leftover files from compaction or write job boundaries receive the same factor.

Polars already fetches file sizes from `object_store` during glob expansion. Each `ObjectMeta` returned by the listing has a `.size` field. Polars uses it only to filter out empty objects and then discards it. Neither `ScanSources` nor `FileInfo` carry file sizes, so they never reach cudf-polars.

Once per-file sizes are available in the `Scan` IR, replace the uniform factor in `scan_partition_plan` with greedy binning: walk files in order, accumulate estimated decompressed size, and emit a task when the cumulative size exceeds `target_partition_size`. Files larger than `target_partition_size` are split at row group boundaries as today. For datasets with uniform file sizes this produces the same result as the current approach. When listing sizes are not available (paths given explicitly rather than via glob), fall back to the current mean-based approach.

## Related

- #22734 file size is in the same `ObjectMeta` struct and could be surfaced alongside etag/last-modified

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.