[PERF]: Use object metadata from polars `Scan` node in cudf-polars
Open
cudf-polars
cuIO
feature request
Performance
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
https://github.com/pola-rs/polars/issues/27828 proposes propagating the object metadata from that polars' `Scan `node to the IR that cudf-polars receives. When available, we should be able to
1. avoid a `HEAD` request per file to discover the object size (currently performed by kvikio)
2. implement etag-based caching of (parts of) files, for both data and metadata
Contributor guide
Assessment
This issue has not been assessed yet.