NVIDIA / NVIDIA/cudf

[FEA]: Use cloud provider credential options in cudf-polars

Open
#19,757 0 comments 2 reactions 0 assignees View on GitHub
cudf-polars feature request Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**

Currently, it's possible to use cudf-polars to read from a private S3 bucket via kvikio, but it requires some changes to the polars. Typically, polars users would write their query using [Cloud authentication](https://docs.pola.rs/user-guide/io/cloud-storage/#cloud-authentication) via a `storage_options` dict or a `CredentialProvider` class

```python
df = pl.scan_parquet(..., credential_provider=pl.CredentialProviderAWS(...))
```

But this will fail with cudf-polars, which uses kvikio for S3 IO. You would need to specify the environment variables ahead of time:

```
# export configuration for kvikio
export AWS_DEFAULT_REGION=...
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
```

And then write the query:

```python
In [1]: import polars as pl

# pl.CredentialProviderAWS isn't used, but is required for zero-code change
# if executing with the CPU engine
In [2]: df = pl.scan_parquet("s3://kvikiobench-33622/data/parquet/timeseries.parquet/part.0.parquet", credential_provider=pl.CredentialProviderAWS(profile_name="default", region_name="us-east-1"))

In [3]: engine = pl.GPUEngine(raise_on_fail=True, executor="streaming")

In [4]: df.head().collect(engine=engine)
Out[4]:
shape: (5, 5)
┌─────────┬──────┬───────────┬───────────┬─────────────────────┐
│ name ┆ id ┆ x ┆ y ┆ timestamp │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ i64 ┆ f64 ┆ f64 ┆ datetime[ns] │
╞═════════╪══════╪═══════════╪═══════════╪═════════════════════╡
│ Alice ┆ 1017 ┆ -0.494019 ┆ 0.946524 ┆ 2000-01-31 00:00:00 │
│ Dan ┆ 1007 ┆ -0.915636 ┆ -0.408707 ┆ 2000-01-31 00:00:10 │
│ Sarah ┆ 1015 ┆ 0.092606 ┆ -0.017722 ┆ 2000-01-31 00:00:20 │
│ Norbert ┆ 1004 ┆ -0.511775 ┆ 0.934027 ┆ 2000-01-31 00:00:30 │
│ Ingrid ┆ 1004 ┆ 0.955055 ┆ 0.851096 ┆ 2000-01-31 00:00:40 │
└─────────┴──────┴───────────┴───────────┴─────────────────────┘
```

**Describe the solution you'd like**

Ideally, cudf-polars would extract the auth information from the `Scan` node and translate it to what kvikio is expecting. Then you just specify your auth like you would with (CPU) polars and don't have to worry about the implementation detail that is kvikio.

**Describe alternatives you've considered**

The current workaround is show above: specify the credentials in both places.

**Additional context**
Add any other context, code examples, or references to existing implementations about the feature request here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.