[FEA]: Use cloud provider credential options in cudf-polars
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
Currently, it's possible to use cudf-polars to read from a private S3 bucket via kvikio, but it requires some changes to the polars. Typically, polars users would write their query using [Cloud authentication](https://docs.pola.rs/user-guide/io/cloud-storage/#cloud-authentication) via a `storage_options` dict or a `CredentialProvider` class
```python
df = pl.scan_parquet(..., credential_provider=pl.CredentialProviderAWS(...))
```
But this will fail with cudf-polars, which uses kvikio for S3 IO. You would need to specify the environment variables ahead of time:
```
# export configuration for kvikio
export AWS_DEFAULT_REGION=...
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
```
And then write the query:
```python
In [1]: import polars as pl
# pl.CredentialProviderAWS isn't used, but is required for zero-code change
# if executing with the CPU engine
In [2]: df = pl.scan_parquet("s3://kvikiobench-33622/data/parquet/timeseries.parquet/part.0.parquet", credential_provider=pl.CredentialProviderAWS(profile_name="default", region_name="us-east-1"))
In [3]: engine = pl.GPUEngine(raise_on_fail=True, executor="streaming")
In [4]: df.head().collect(engine=engine)
Out[4]:
shape: (5, 5)
┌─────────┬──────┬───────────┬───────────┬─────────────────────┐
│ name ┆ id ┆ x ┆ y ┆ timestamp │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ i64 ┆ f64 ┆ f64 ┆ datetime[ns] │
╞═════════╪══════╪═══════════╪═══════════╪═════════════════════╡
│ Alice ┆ 1017 ┆ -0.494019 ┆ 0.946524 ┆ 2000-01-31 00:00:00 │
│ Dan ┆ 1007 ┆ -0.915636 ┆ -0.408707 ┆ 2000-01-31 00:00:10 │
│ Sarah ┆ 1015 ┆ 0.092606 ┆ -0.017722 ┆ 2000-01-31 00:00:20 │
│ Norbert ┆ 1004 ┆ -0.511775 ┆ 0.934027 ┆ 2000-01-31 00:00:30 │
│ Ingrid ┆ 1004 ┆ 0.955055 ┆ 0.851096 ┆ 2000-01-31 00:00:40 │
└─────────┴──────┴───────────┴───────────┴─────────────────────┘
```
**Describe the solution you'd like**
Ideally, cudf-polars would extract the auth information from the `Scan` node and translate it to what kvikio is expecting. Then you just specify your auth like you would with (CPU) polars and don't have to worry about the implementation detail that is kvikio.
**Describe alternatives you've considered**
The current workaround is show above: specify the credentials in both places.
**Additional context**
Add any other context, code examples, or references to existing implementations about the feature request here.
Contributor guide
Assessment
This issue has not been assessed yet.