AWS S3 HTTP 400 Bad Request Error - Exception Message Improvement
- Dominant language
- C++
- Stars
- 60
- Forks
- 100
- Avg merge
- 1h 50m
- Merged PRs (30d)
- 25
Description
Occasionally (for some reason which we do not know), we get an HTTP 400 error when using DuckDB with httpfs extension and connecting to AWS S3.
```
HTTP Error: HTTP GET error on 'https://bucket.s3.us-east-1.amazonaws.com/path/year%3D2025/month%3D7/file2.snappy.parquet' (HTTP 400) Bad Request - this can be caused by the S3 region being set incorrectly. * Provided region is "us-east-1"
```
Here is the error handling in the code:
https://github.com/duckdb/duckdb-httpfs/blob/main/extension/httpfs/s3fs.cpp#L772
In AWS S3, `400 Bad Request` can be for any number of reasons:
https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html#ErrorCodeList
I'm looking for an improvement to the error message that gets logged to indicate the reason for the `HTTP 400` so that further troubleshooting can be performed.
In our use case, we are using DuckDB secret manager for connecting and authenticating, and we set the region and endpoint explicitly...
```sql
CREATE TEMPORARY SECRET s_123 (
TYPE S3,
KEY_ID '...',
SECRET '...',
SESSION_TOKEN '...',
REGION 'us-east-1',
ENDPOINT 's3.us-east-1.amazonaws.com',
SCOPE 'https://bucket.s3.us-east-1.amazonaws.com/path/',
URL_STYLE 'vhost',
USE_SSL true
);
```
The request was made to multiple parquet files, to add more context:
```sql
SELECT * FROM read_parquet(['s3://bucket/path/year%3D2025/month%3D7/file1.snappy.parquet','s3://bucket/path/year%3D2025/month%3D7/file2.snappy.parquet','s3://bucket/path/year%3D2025/month%3D7/file3.snappy.parquet']);
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.