NVIDIA / NVIDIA/cudf

[FEA] Propagate relevant `storage_options` to KvikIO when reading from s3 with cuDF and kvikio s3 is enabled

Open
#19,758 0 comments 1 reaction 0 assignees View on GitHub
feature request Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**
The current approach to using KviKIO enabled s3 reads is to export all AWS specific credentials/auth options via env variables for kvikio and the `storage_options` provided aren't used/mapped to the read.

**Describe the solution you'd like**
Ideally cuDF could map `storage_options` into kvikio compliant options and propagate them down to the kvikio layer if kvikio s3 reads are enabled.

**Describe alternatives you've considered**
Current solution requires setting the options in a couple of places (storage_options+env vars) to fully leverage kvikio s3 + cudf reads.

**Additional context**
Related to #19757 (and possibly the solution is similar) but this request specific to the cuDF api.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.