duckdb / duckdb/duckdb-httpfs

Reading S3 object metadata

Open
#38 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
60
Forks
100
Avg merge
1h 50m
Merged PRs (30d)
25

Description

It would be nice to be able to query [S3 object metadata](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html) directly.

Maybe it could use the `parquet_kv_metadata` as an interface?

```sql
SELECT * FROM parquet_kv_metadata('s3://bucket/file.parquet');
```
```
file_name,key,value
's3://bucket/file.parquet', pandas,
's3://bucket/file.parquet', ARROW:schema,
's3://bucket/file.parquet', s3_metadata, {"ETag": "awefawefawef"}
```

(apologies if this is already possible, I couldn't find anything about it in the docs)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the existing parquet_kv_metadata interface and the AWS S3 object metadata documentation linked in the issue. Determine how S3 metadata could be exposed through the suggested SQL query, then verify that querying an S3 Parquet file returns its object metadata alongside the existing key-value metadata.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, cpp
Domain
cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.