duckdb / duckdb/duckdb-httpfs

HTTPFS COPY (PythonClient) To S3 silently succeeds with invalid credentials — no error raised

Open
#304 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
60
Forks
100
Avg merge
1h 50m
Merged PRs (30d)
25

Description

## What happens?

When using `COPY ... TO 's3://...'` with invalid S3 credentials (wrong secret key), the command completes without raising any exception. No data is written to S3, but the caller has no way to know the write failed.

This causes silent data loss in automated pipelines where the return/exception from `COPY TO` is the only signal of success.

## To Reproduce

```sql
LOAD httpfs;

-- Create secret with a valid key ID but WRONG secret
CREATE SECRET output_secret (
TYPE S3,
KEY_ID 'valid_access_key_id',
SECRET 'wrong_secret_key',
REGION 'us-east-1',
SCOPE 's3://my-bucket/'
);

CREATE TABLE test AS SELECT 'hello' AS col;

-- This succeeds without error — but nothing is written to S3
COPY test TO 's3://my-bucket/test.csv.gz' (FORMAT CSV, COMPRESSION GZIP);
```

## Expected behavior

The `COPY TO` command should raise an error (e.g., `IOException` or `HTTPException`) when S3 returns `SignatureDoesNotMatch` or `AccessDenied` during the multipart upload.

## Context

- Verified with `aws sts get-caller-identity` that the same wrong secret correctly returns `SignatureDoesNotMatch`
- Verified with `aws s3api put-object` (boto3) that the same wrong secret correctly raises `SignatureDoesNotMatch`
- The issue appears to be in the httpfs S3 write path — authentication errors from multipart upload initiation or part upload are swallowed silently
- Related to duckdb/duckdb#9376 and duckdb/duckdb#14877 (silent multipart upload failures), but different root cause: those were about compression handling, this is about auth errors being suppressed

## OS

macOS (also reproducible on Linux ARM64 / Graviton)

## DuckDB Version

Tested on 1.1.x and 1.5.1 (latest) — bug present in both.

## DuckDB Client

Python API

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the failure through the Python API using the SQL example and invalid S3 credentials. Trace the httpfs S3 write path around multipart upload initiation and part upload, then verify that SignatureDoesNotMatch or AccessDenied is propagated and that COPY TO raises an exception instead of succeeding silently.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, cpp, python
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.