duckdb / duckdb/duckdb-httpfs

Cloudflare R2: misleading error when AWS_SESSION_TOKEN is set

Open
#254 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
60
Forks
100
Avg merge
1h 50m
Merged PRs (30d)
25

Description

### Description:

When running DuckDB inside AWS Lambda, the `AWS_SESSION_TOKEN` environment variable (injected automatically by the Lambda
runtime for IAM role credentials) is picked up by `httpfs` and sent as the `X-Amz-Security-Token` header on S3 requests —
even when using an explicit `CREATE SECRET` with static credentials.

This causes HTTP 400 errors when writing to S3-compatible stores like Cloudflare R2 that don't support STS session
tokens. The same code works from AWS ECS or other situations where `AWS_SESSION_TOKEN` isn't exposed as an env var.

### Reproduction:

1. Create an AWS Lambda function with a DuckDB binary. Or alternatively launch duckdb w/ `AWS_SESSION_TOKEN` set to a garbage value.
```bash
AWS_SESSION_TOKEN=garbage duckdb
```
2. Configure an explicit secret with static R2 credentials:
```sql
CREATE SECRET r2_secret (TYPE r2, KEY_ID '...', SECRET '...', ACCOUNT_ID '...');
```
3. Attempt any write to R2:
```sql
COPY (SELECT 1) TO 'r2://bucket/test.parquet' (FORMAT PARQUET);
```

4. Result: `HTTP Error: Unable to connect to URL https://....r2.cloudflarestorage.com/...: Bad Request (HTTP code 400)`

Expected behavior: An explicit `CREATE SECRET` with `KEY_ID` and `SECRET` should not inherit the `AWS_SESSION_TOKEN` from the
environment.

### Workaround:

Before interacting with R2 clear the `httpfs` extensions copy of the session token via:

```sql
SET s3_session_token = '';
COPY (SELECT 1) TO 'r2://bucket/test.parquet' (FORMAT PARQUET); -- Works!
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the failure with AWS_SESSION_TOKEN=garbage, the CREATE SECRET statement, and the R2 COPY command. Read the httpfs credential and session-token handling around explicit secrets, then verify that the same COPY succeeds without inheriting the environment token and that the s3_session_token workaround is no longer needed.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, cpp, sql
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.