NVIDIA / NVIDIA/cudf

[PERF]: Avoid discarding prefetched parquet metadata for scans that mix cached and uncached metadata for paths

Open
#23,647 1 comment 0 reactions 0 assignees View on GitHub
cudf-polars Performance
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

[Optional] In cases where a single scan reads from both local and remote paths, since we've already read and cached the parquet info, is there a good reason to throw it away if some path has not been cached (is not remote)? Can we not apply the ones we have anyway, e.g., `cached = [cached_parquet_info_map.get(path, None) for path in scan.paths]`, with appropriate subsequent handling of `None` by splitting the scan into two reads _or_ adding a signal to libcudf to read a footer on-demand? Or could we at least skip the remote metadata fetch for paths that only appear in such scans, where we know we'd be throwing it away?

_Originally posted by @igorpeshansky in https://github.com/NVIDIA/cudf/pull/23558#discussion_r3772353541_

Contributor guide

Open the contributing guide

Research direction

Start by tracing cached_parquet_info_map and the scan path handling described in the issue, then review the linked discussion in PR 23558. Determine how mixed cached and uncached paths are currently handled and whether retaining available metadata or skipping unnecessary remote fetches is feasible. Done means the chosen behavior is implemented without discarding usable metadata and is covered by appropriate validation.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.