[Feature request] Add support to Read Athena Results from S3 Using DuckDB
- Dominant language
- Rust
- Stars
- 20.8k
- Forks
- 2.1k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 181
Description
Currently, Athena Drive relies solely on the AWS API, which is great for access control management. However, when handling large query results in streaming mode, performance can be significantly impacted due to the REST API's limitation of returning a maximum of 1,000 rows per request.
It would be beneficial to have an optional mode to read the generated CSV result directly from S3 using DuckDB, which is extremely fast and would allow fetching the entire result with a single request instead of iterating over multiple paginated responses.
This would not replace the default behavior but serve as an opt-in alternative for scenarios where performance is a concern.
If this makes sense, I'm happy to contribute with a PR for this feature. Let me know your thoughts!
Contributor guide
Assessment
This issue has not been assessed yet.