[PERF]: Avoid discarding prefetched parquet metadata for scans that mix cached and uncached metadata for paths
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
[Optional] In cases where a single scan reads from both local and remote paths, since we've already read and cached the parquet info, is there a good reason to throw it away if some path has not been cached (is not remote)? Can we not apply the ones we have anyway, e.g., `cached = [cached_parquet_info_map.get(path, None) for path in scan.paths]`, with appropriate subsequent handling of `None` by splitting the scan into two reads _or_ adding a signal to libcudf to read a footer on-demand? Or could we at least skip the remote metadata fetch for paths that only appear in such scans, where we know we'd be throwing it away?
_Originally posted by @igorpeshansky in https://github.com/NVIDIA/cudf/pull/23558#discussion_r3772353541_
Contributor guide
Research direction
Start by tracing cached_parquet_info_map and the scan path handling described in the issue, then review the linked discussion in PR 23558. Determine how mixed cached and uncached paths are currently handled and whether retaining available metadata or skipping unnecessary remote fetches is feasible. Done means the chosen behavior is implemented without discarding usable metadata and is covered by appropriate validation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100