[httpfs] enable_global_s3_configuration = false still ingests AWS_* env credentials, unredacted
- Dominant language
- C++
- Stars
- 60
- Forks
- 100
- Avg merge
- 1h 50m
- Merged PRs (30d)
- 25
Description
Loading httpfs reads AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN from the environment into the legacy s3_* settings, where any caller can read them in plaintext from duckdb_settings() or current_setting(). allow_unredacted_secrets is false (the default) and has no effect on these settings.
Also, there is a documented setting called enable_global_s3_configuration, which it says: _Automatically fetch AWS credentials from environment variables._ We have set this to false, and yet the credentials still get pulled in from the environment. Please see the second example below.
We run DuckDB inside AWS Lambda and expose a query interface to end users. Lambda's runtime injects those three variables into every invocation. We don't set them and can't remove them, so there is no way for us to `LOAD httpfs` without publishing our execution role's live credentials to anyone who can run a SELECT. The [legacy auth docs](https://duckdb.org/docs/current/core_extensions/httpfs/s3api_legacy_authentication) advise avoiding this scheme, but the ingestion is automatic.
Note in the transcript below that duckdb_secrets() correctly redacts secret and session_token, while the identical values remain in plaintext in duckdb_settings() — and that creating a CREDENTIAL_CHAIN secret does not clear the legacy settings.
Ideally: make enable_global_s3_configuration = false actually prevent the ingestion, and redact the legacy s3_* credential settings unless allow_unredacted_secrets = true. The first is the bug; the second is defense-in-depth, given that the Secrets Manager already redacts the identical values (and that a CREDENTIAL_CHAIN secret does not clear the legacy settings, so the documented migration path doesn't remediate this either).
Reproduced on 1.5.5 (latest stable), CLI, macOS. Fake credentials below.
## To reproduce
```
(base) kwinkler@Kevins-MacBook-Air Cirro-backend % duckdb
DuckDB v1.5.5 (Variegata)
Enter ".help" for usage hints.
memory D select name, value from duckdb_settings() where name in ('s3_access_key_id', 's3_secret_access_key', 's3_session_token');
┌─────────┬─────────┐
│ name │ value │
│ varchar │ varchar │
└─────────┴─────────┘
0 rows
memory D select * from duckdb_settings() where name like '%redact%';
┌──────────────────────────┬─────────┬───────────────────────────────────┬────────────┬─────────┬───────────┐
│ name │ value │ description │ input_type │ scope │ aliases │
│ varchar │ varchar │ varchar │ varchar │ varchar │ varchar[] │
├──────────────────────────┼─────────┼───────────────────────────────────┼────────────┼─────────┼───────────┤
│ allow_unredacted_secrets │ false │ Allow printing unredacted secrets │ BOOLEAN │ GLOBAL │ [] │
└──────────────────────────┴─────────┴───────────────────────────────────┴────────────┴─────────┴───────────┘
memory D load httpfs;
memory D select name, value from duckdb_settings() where name in ('s3_access_key_id', 's3_secret_access_key', 's3_session_token');
┌──────────────────────┬───────────────┐
│ name │ value │
│ varchar │ varchar │
├──────────────────────┼───────────────┤
│ s3_access_key_id │ fakeaccesskey │
│ s3_secret_access_key │ thisisfake │
│ s3_session_token │ faketoken │
└──────────────────────┴───────────────┘
memory D CREATE SECRET IF NOT EXISTS aws_cred (TYPE S3, PROVIDER CREDENTIAL_CHAIN);
WARNING:
Set region explicitly using REGION 'us-east-1' in your CREATE SECRET statement, adding a region to your profile in ~/.aws/config or configure the AWS_REGION or AWS_DEFAULT_REGION environment variables.
┌─────────┐
│ Success │
│ boolean │
├─────────┤
│ true │
└─────────┘
memory D select name, value from duckdb_settings() where name in ('s3_access_key_id', 's3_secret_access_key', 's3_session_token');
┌──────────────────────┬───────────────┐
│ name │ value │
│ varchar │ varchar │
├──────────────────────┼───────────────┤
│ s3_access_key_id │ fakeaccesskey │
│ s3_secret_access_key │ thisisfake │
│ s3_session_token │ faketoken │
└──────────────────────┴───────────────┘
memory D select * from duckdb_secrets();
┌──────────┬─────────┬──────────────────┬────────────┬─────────┬───────────────────────────────┬─────────────────────────────────────────────────────────────────────────────────────────┐
│ name │ type │ provider │ persistent │ storage │ scope │ secret_string │
│ varchar │ varchar │ varchar │ boolean │ varchar │ varchar[] │ varchar │
├──────────┼─────────┼──────────────────┼────────────┼─────────┼───────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────┤
│ aws_cred │ s3 │ credential_chain │ false │ memory │ ['s3://', 's3n://', 's3a://'] │ name=aws_cred;type=s3;provider=credential_chain;serializable=true;scope=s3://,s3n://,s3 │
│ │ │ │ │ │ │ a://;endpoint=s3.amazonaws.com;key_id=fakeaccesskey;secret=redacted;session_token=redac │
│ │ │ │ │ │ │ ted │
└──────────┴─────────┴──────────────────┴────────────┴─────────┴───────────────────────────────┴─────────────────────────────────────────────────────────────────────────────────────────┘
memory D
```
Note: I thought enable_global_s3_configuration set to false might be the solution, but the env creds still were ingested into the DuckDB environment.
```
(base) kwinkler@Kevins-MacBook-Air Cirro-backend % duckdb
DuckDB v1.5.5 (Variegata)
Enter ".help" for usage hints.
memory D SET enable_global_s3_configuration=false;
memory D load httpfs;
memory D select name, value from duckdb_settings() where name in ('s3_access_key_id', 's3_secret_access_key', 's3_session_token');
┌──────────────────────┬───────────────┐
│ name │ value │
│ varchar │ varchar │
├──────────────────────┼───────────────┤
│ s3_access_key_id │ fakeaccesskey │
│ s3_secret_access_key │ thisisfake │
│ s3_session_token │ faketoken │
└──────────────────────┴───────────────┘
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the credential ingestion with LOAD httpfs and enable_global_s3_configuration=false, then inspect the httpfs settings and credential-handling entry points for s3_* values. Done means disabling global S3 configuration prevents AWS_* ingestion, and legacy credential settings are redacted unless allow_unredacted_secrets is true; verify with duckdb_settings() and duckdb_secrets().
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- cloud, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100