duckdb / duckdb/duckdb-httpfs

Implement aws s3 StartAfter optimization

Open
#230 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
60
Forks
100
Avg merge
1h 50m
Merged PRs (30d)
25

Description

# Context
As a user of duckdb and aws s3, I often find myself wanting to use the "StartAfter" key along with existing prefix globbing features of duckdb. This helps reduce the scan time on large s3 buckets.

For example, let's assume we have a process streams some data directly to a series of s3 files:

```raw
s3://my-bucket/dataset_a_202512.csv
s3://my-bucket/dataset_a_202601.csv
s3://my-bucket/dataset_a_202602.csv
```

the current process would allow us to compute a prefix and scan an entire year, year-month, etc. per call, so if we only wanted files this year
```sql
select * from 's3://my-bucket/dataset_a_2026*.csv`
```
works well.

However, if I also have a last read time of some sort, I'd like to utilize list_object_v2's [startAfter parameter](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#API_ListObjectsV2_RequestSyntax)

This would be some api call that looks like this:
```raw
GET /?list-type=2
&prefix=dataset_a_
&start-after=dataset_a_202601.csv
&encoding-type=url
HTTP/1.1
Host: my-bucket.s3.us-west-2.amazonaws.com
```

## Implementation

I'm not sure on exactly how this should be implmented. While this seems a natural feature add of httpfs, I'm not sure if this s3 specific feature is intended for use or not here. Additionally, even if we were to implement it, there's questions on how this would actually show up in SQL. It's likely not a good fit for the existing glob syntax since this would end up overcomplicating it and making it non-standard (IMO).

The two other routes forward would be to:
1. Add StartAfter to all httpfs enabled apis (read_parquet, read_csv, etc)
2. Create a new 's3_list' of some sort. While this may work, I'd be worried about how well this would interface with other existing workflows.

Would be interested in here other's thoughts

Thanks

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the httpfs S3 listing path and the list_object_v2 StartAfter parameter described in the issue. Compare how the proposed read_parquet/read_csv APIs or an s3_list entry point could expose it; the issue is done when one supported SQL-facing design is agreed and its behavior for prefix scans is defined.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.