duckdb / duckdb/duckdb-httpfs

AWS S3 HTTP 400 Bad Request Error - Exception Message Improvement

Open
#87 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
60
Forks
100
Avg merge
1h 50m
Merged PRs (30d)
25

Description

Occasionally (for some reason which we do not know), we get an HTTP 400 error when using DuckDB with httpfs extension and connecting to AWS S3.

```
HTTP Error: HTTP GET error on 'https://bucket.s3.us-east-1.amazonaws.com/path/year%3D2025/month%3D7/file2.snappy.parquet' (HTTP 400) Bad Request - this can be caused by the S3 region being set incorrectly. * Provided region is "us-east-1"
```

Here is the error handling in the code:
https://github.com/duckdb/duckdb-httpfs/blob/main/extension/httpfs/s3fs.cpp#L772

In AWS S3, `400 Bad Request` can be for any number of reasons:
https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html#ErrorCodeList

I'm looking for an improvement to the error message that gets logged to indicate the reason for the `HTTP 400` so that further troubleshooting can be performed.

In our use case, we are using DuckDB secret manager for connecting and authenticating, and we set the region and endpoint explicitly...

```sql
CREATE TEMPORARY SECRET s_123 (
TYPE S3,
KEY_ID '...',
SECRET '...',
SESSION_TOKEN '...',
REGION 'us-east-1',
ENDPOINT 's3.us-east-1.amazonaws.com',
SCOPE 'https://bucket.s3.us-east-1.amazonaws.com/path/',
URL_STYLE 'vhost',
USE_SSL true
);
```

The request was made to multiple parquet files, to add more context:

```sql
SELECT * FROM read_parquet(['s3://bucket/path/year%3D2025/month%3D7/file1.snappy.parquet','s3://bucket/path/year%3D2025/month%3D7/file2.snappy.parquet','s3://bucket/path/year%3D2025/month%3D7/file3.snappy.parquet']);
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.