Support storage_options in Parquet metadata reads (cudf.io.parquet.read_parquet_metadata)
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
## Summary
`cudf.io.parquet.read_parquet_metadata` does not accept `storage_options`, unlike `cudf.read_parquet`. This prevents callers from reading footer metadata from authenticated S3-compatible/object-store inputs using the same credentials and endpoint configuration as the data read.
This also affects users of the RAPIDS 26.08 bulk helper `pylibcudf.io.parquet_metadata.read_parquet_footers`: `pylibcudf.io.SourceInfo` accepts sources, but there is no `storage_options` parameter to resolve authenticated remote paths. The available workaround is to depend on private cuDF I/O path-resolution helpers or materialize sources through fsspec.
## Reproducer
```python
import cudf
cudf.io.parquet.read_parquet_metadata(
"s3://bucket/data.parquet",
storage_options={
"key": "...",
"secret": "...",
"client_kwargs": {"endpoint_url": "https://s3-compatible.example"},
},
)
```
On cuDF 26.08.00 this raises:
```
TypeError: read_parquet_metadata() got an unexpected keyword argument 'storage_options'
```
The current signature is:
```python
read_parquet_metadata(filepath_or_buffer)
```
## Expected behavior
Accept `storage_options=None` and resolve remote inputs consistently with `cudf.read_parquet`, while retaining bulk input support and input ordering for `read_parquet_footers`/metadata reads.
## Environment
- cuDF 26.08.00
- pylibcudf 26.08 bindings
- NVIDIA NeMo Curator nightly 2026-08-27
This is needed to replace per-file PyArrow footer reads in NeMo Curator's KMeans/Pairwise grouping without losing support for authenticated remote and S3-compatible datasets.
Contributor guide
Research direction
Start at cudf.io.parquet.read_parquet_metadata and pylibcudf.io.parquet_metadata.read_parquet_footers, then compare how cudf.read_parquet handles storage_options. Done means authenticated remote inputs accept storage_options while bulk inputs and their ordering remain supported; add or update the relevant tests found alongside these entry points.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100