Implement aws s3 StartAfter optimization
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 60
- Forks
- 100
- Avg merge
- 1h 50m
- Merged PRs (30d)
- 25
Description
# Context
As a user of duckdb and aws s3, I often find myself wanting to use the "StartAfter" key along with existing prefix globbing features of duckdb. This helps reduce the scan time on large s3 buckets.
For example, let's assume we have a process streams some data directly to a series of s3 files:
```raw
s3://my-bucket/dataset_a_202512.csv
s3://my-bucket/dataset_a_202601.csv
s3://my-bucket/dataset_a_202602.csv
```
the current process would allow us to compute a prefix and scan an entire year, year-month, etc. per call, so if we only wanted files this year
```sql
select * from 's3://my-bucket/dataset_a_2026*.csv`
```
works well.
However, if I also have a last read time of some sort, I'd like to utilize list_object_v2's [startAfter parameter](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#API_ListObjectsV2_RequestSyntax)
This would be some api call that looks like this:
```raw
GET /?list-type=2
&prefix=dataset_a_
&start-after=dataset_a_202601.csv
&encoding-type=url
HTTP/1.1
Host: my-bucket.s3.us-west-2.amazonaws.com
```
## Implementation
I'm not sure on exactly how this should be implmented. While this seems a natural feature add of httpfs, I'm not sure if this s3 specific feature is intended for use or not here. Additionally, even if we were to implement it, there's questions on how this would actually show up in SQL. It's likely not a good fit for the existing glob syntax since this would end up overcomplicating it and making it non-standard (IMO).
The two other routes forward would be to:
1. Add StartAfter to all httpfs enabled apis (read_parquet, read_csv, etc)
2. Create a new 's3_list' of some sort. While this may work, I'd be worried about how well this would interface with other existing workflows.
Would be interested in here other's thoughts
Thanks
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the httpfs S3 listing path and the list_object_v2 StartAfter parameter described in the issue. Compare how the proposed read_parquet/read_csv APIs or an s3_list entry point could expose it; the issue is done when one supported SQL-facing design is agreed and its behavior for prefix scans is defined.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100