NVIDIA / NVIDIA/cudf

Support storage_options in Parquet metadata reads (cudf.io.parquet.read_parquet_metadata)

Open
#23,899 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

## Summary

`cudf.io.parquet.read_parquet_metadata` does not accept `storage_options`, unlike `cudf.read_parquet`. This prevents callers from reading footer metadata from authenticated S3-compatible/object-store inputs using the same credentials and endpoint configuration as the data read.

This also affects users of the RAPIDS 26.08 bulk helper `pylibcudf.io.parquet_metadata.read_parquet_footers`: `pylibcudf.io.SourceInfo` accepts sources, but there is no `storage_options` parameter to resolve authenticated remote paths. The available workaround is to depend on private cuDF I/O path-resolution helpers or materialize sources through fsspec.

## Reproducer

```python
import cudf

cudf.io.parquet.read_parquet_metadata(
"s3://bucket/data.parquet",
storage_options={
"key": "...",
"secret": "...",
"client_kwargs": {"endpoint_url": "https://s3-compatible.example"},
},
)
```

On cuDF 26.08.00 this raises:

```
TypeError: read_parquet_metadata() got an unexpected keyword argument 'storage_options'
```

The current signature is:

```python
read_parquet_metadata(filepath_or_buffer)
```

## Expected behavior

Accept `storage_options=None` and resolve remote inputs consistently with `cudf.read_parquet`, while retaining bulk input support and input ordering for `read_parquet_footers`/metadata reads.

## Environment

- cuDF 26.08.00
- pylibcudf 26.08 bindings
- NVIDIA NeMo Curator nightly 2026-08-27

This is needed to replace per-file PyArrow footer reads in NeMo Curator's KMeans/Pairwise grouping without losing support for authenticated remote and S3-compatible datasets.

Contributor guide

Open the contributing guide

Research direction

Start at cudf.io.parquet.read_parquet_metadata and pylibcudf.io.parquet_metadata.read_parquet_footers, then compare how cudf.read_parquet handles storage_options. Done means authenticated remote inputs accept storage_options while bulk inputs and their ordering remain supported; add or update the relevant tests found alongside these entry points.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.