[FEA] Support passing access credentials of remote files as function parameters
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
Currently, remote file access in `libcudf` is performed using `KvikIO`'s utility function [open](https://github.com/rapidsai/cudf/blob/main/cpp/src/io/utilities/datasource.cpp#L364), which has the convenience of automatically inferring the endpoint type (such as S3 private, S3 public, S3 presigned URL, WebHDFS, generic HTTP/HTTPS), but has the limitation that the access credentials, if needed, can only be specified via environment variables, which applies to all data sources indiscriminately for the running process.
Ideally, the access credentials should be able to get passed to `libcudf` (and also `pylibcudf`, `cuDF` and `KvikIO`'s open interface) as function parameters on a per-data-source basis, which should take higher precedence than the environment variables if both are present.
**Describe the solution you'd like**
- Expose a remote file credential class in C++ and Python API
- Update all I/O interfaces to accept the optional credential object
**Describe alternatives you've considered**
- Support an optional `file-credential-mapping.cfg`, where users specify which remote source uses which credentials in a plain text file, and an environment variable `CUDF_REMOTE_FILE_CREDENTIAL_MAPPING` is used to specify the configure file path. This method can still enable per-data-source authentication while keeping the code changes to a minimum.
**Additional context**
This general feature to be able to programmatically pass credentials will also help to address https://github.com/rapidsai/cudf/issues/19758 and https://github.com/rapidsai/cudf/issues/19757.
Contributor guide
Assessment
This issue has not been assessed yet.