HTTPFS COPY (PythonClient) To S3 silently succeeds with invalid credentials — no error raised
- Dominant language
- C++
- Stars
- 60
- Forks
- 100
- Avg merge
- 1h 50m
- Merged PRs (30d)
- 25
Description
## What happens?
When using `COPY ... TO 's3://...'` with invalid S3 credentials (wrong secret key), the command completes without raising any exception. No data is written to S3, but the caller has no way to know the write failed.
This causes silent data loss in automated pipelines where the return/exception from `COPY TO` is the only signal of success.
## To Reproduce
```sql
LOAD httpfs;
-- Create secret with a valid key ID but WRONG secret
CREATE SECRET output_secret (
TYPE S3,
KEY_ID 'valid_access_key_id',
SECRET 'wrong_secret_key',
REGION 'us-east-1',
SCOPE 's3://my-bucket/'
);
CREATE TABLE test AS SELECT 'hello' AS col;
-- This succeeds without error — but nothing is written to S3
COPY test TO 's3://my-bucket/test.csv.gz' (FORMAT CSV, COMPRESSION GZIP);
```
## Expected behavior
The `COPY TO` command should raise an error (e.g., `IOException` or `HTTPException`) when S3 returns `SignatureDoesNotMatch` or `AccessDenied` during the multipart upload.
## Context
- Verified with `aws sts get-caller-identity` that the same wrong secret correctly returns `SignatureDoesNotMatch`
- Verified with `aws s3api put-object` (boto3) that the same wrong secret correctly raises `SignatureDoesNotMatch`
- The issue appears to be in the httpfs S3 write path — authentication errors from multipart upload initiation or part upload are swallowed silently
- Related to duckdb/duckdb#9376 and duckdb/duckdb#14877 (silent multipart upload failures), but different root cause: those were about compression handling, this is about auth errors being suppressed
## OS
macOS (also reproducible on Linux ARM64 / Graviton)
## DuckDB Version
Tested on 1.1.x and 1.5.1 (latest) — bug present in both.
## DuckDB Client
Python API
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the failure through the Python API using the SQL example and invalid S3 credentials. Trace the httpfs S3 write path around multipart upload initiation and part upload, then verify that SignatureDoesNotMatch or AccessDenied is propagated and that COPY TO raises an exception instead of succeeding silently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, cpp, python
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100