Cloudflare R2: misleading error when AWS_SESSION_TOKEN is set
- Dominant language
- C++
- Stars
- 60
- Forks
- 100
- Avg merge
- 1h 50m
- Merged PRs (30d)
- 25
Description
### Description:
When running DuckDB inside AWS Lambda, the `AWS_SESSION_TOKEN` environment variable (injected automatically by the Lambda
runtime for IAM role credentials) is picked up by `httpfs` and sent as the `X-Amz-Security-Token` header on S3 requests —
even when using an explicit `CREATE SECRET` with static credentials.
This causes HTTP 400 errors when writing to S3-compatible stores like Cloudflare R2 that don't support STS session
tokens. The same code works from AWS ECS or other situations where `AWS_SESSION_TOKEN` isn't exposed as an env var.
### Reproduction:
1. Create an AWS Lambda function with a DuckDB binary. Or alternatively launch duckdb w/ `AWS_SESSION_TOKEN` set to a garbage value.
```bash
AWS_SESSION_TOKEN=garbage duckdb
```
2. Configure an explicit secret with static R2 credentials:
```sql
CREATE SECRET r2_secret (TYPE r2, KEY_ID '...', SECRET '...', ACCOUNT_ID '...');
```
3. Attempt any write to R2:
```sql
COPY (SELECT 1) TO 'r2://bucket/test.parquet' (FORMAT PARQUET);
```
4. Result: `HTTP Error: Unable to connect to URL https://....r2.cloudflarestorage.com/...: Bad Request (HTTP code 400)`
Expected behavior: An explicit `CREATE SECRET` with `KEY_ID` and `SECRET` should not inherit the `AWS_SESSION_TOKEN` from the
environment.
### Workaround:
Before interacting with R2 clear the `httpfs` extensions copy of the session token via:
```sql
SET s3_session_token = '';
COPY (SELECT 1) TO 'r2://bucket/test.parquet' (FORMAT PARQUET); -- Works!
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the failure with AWS_SESSION_TOKEN=garbage, the CREATE SECRET statement, and the R2 COPY command. Read the httpfs credential and session-token handling around explicit secrets, then verify that the same COPY succeeds without inheriting the environment token and that the s3_session_token workaround is no longer needed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, cpp, sql
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100