block / block/buzz

Helm chart cannot select the AWS credential chain for S3 — Pod Identity / IRSA is unreachable

Open
#5,211 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

Component: deploy/charts/buzz (chart 0.1.7) + crates/buzz-relay/src/config.rs
Ref: main

**Describe the bug**
The relay supports resolving S3 credentials from the AWS credential chain, and the
workspace carries a dependency patch specifically so EKS Pod Identity works. But
that path cannot be selected through the Helm chart: a deployment that omits the S3
keys ends up using the literal placeholder credentials buzz_dev / buzz_dev_secret
and fails with opaque S3 403s instead of using the pod's IAM role.

**Steps to reproduce**
Install the chart in the production profile against real AWS S3, with a
secrets.existingSecret that omits BUZZ_S3_ACCESS_KEY / BUZZ_S3_SECRET_KEY, on a
pod whose service account has S3 access via EKS Pod Identity or IRSA.

- **Expected**: the relay resolves credentials from the pod's IAM role.
- **Actual**: it signs requests as buzz_dev and every media/git object operation fails
with 403. On a default install the startup BUZZ_GIT_CONFORMANCE_PROBE makes this
startup-fatal, so readiness never opens — with no log line indicating that placeholder
credentials were substituted.

**Expected behavior**

_Why it should work_

Both storage constructors explicitly branch on empty credentials —
buzz-media/src/storage.rs::MediaStorage::new and
buzz-relay/src/api/git/store.rs::Store::new:

```rust
let creds = match (access_key.is_empty(), secret_key.is_empty()) {
(false, false) => Credentials::new(Some(access_key), Some(secret_key), None, None, None),
(true, true) => Credentials::default(), // env, profile, web-identity/IRSA, container, IMDS
_ => return Err(/* half-configured static deployment */),
};
```
…and the workspace Cargo.toml patches aws-creds to a fork precisely because the
crates.io version "cannot read EKS Pod Identity credentials
(AWS_CONTAINER_CREDENTIALS_FULL_URI + AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE),
which the relay pod on bb-block requires for S3 media + git storage."

So the capability is intended and shipped.

_Why it doesn't_

Two behaviors combine badly.

1. The chart renders the keys as optional secretKeyRefs (templates/deployment.yaml):

```yaml
- name: BUZZ_S3_ACCESS_KEY
valueFrom:
secretKeyRef:
name: {{ include "buzz.envSecretName" . }}
key: BUZZ_S3_ACCESS_KEY
optional: true
```

If the key is absent from the Secret, optional: true means the env var is not set
at all — there is no chart value that renders it as an explicit empty string.

2. The relay substitutes non-empty placeholders for absent vars
(crates/buzz-relay/src/config.rs):

```rust
s3_access_key: std::env::var("BUZZ_S3_ACCESS_KEY").unwrap_or_else(|_| "buzz_dev".to_string()),
s3_secret_key: std::env::var("BUZZ_S3_SECRET_KEY").unwrap_or_else(|_| "buzz_dev_secret".to_string()),
"buzz_dev" is not empty, so MediaStorage::new takes the static branch with
placeholder credentials. Credentials::default() is never reached.
```

The chain branch is only reachable when the env vars are present and empty, since
std::env::var returns Ok("") for a set-but-empty var and unwrap_or_else doesn't fire.

**Version and platform**
Find your version at the bottom of the Settings sidebar. Write "unknown" if you can't determine it.
- Buzz version: unknown - helm chart 0.1.7
- OS: Linux

**Logs / additional context**

_Suggested fix_
Either half resolves it; both together is cleanest.

1. **Relay** — unwrap_or_else(|_| "buzz_dev") → unwrap_or_default(), so an absent var
becomes empty and selects the chain. This also removes the silent-placeholder footgun.
Local MinIO development would need those defaults moved into .env.example /
compose rather than living in code.

2. **Chart** — add an explicit credential-source selector, e.g.
s3.credentialSource: static | chain. On chain, emit BUZZ_S3_ACCESS_KEY: "" and
BUZZ_S3_SECRET_KEY: "" as literal values instead of secretKeyRefs (or omit them
entirely once the relay change lands). templates/_validate.tpl should then accept
s3.bucket + credentialSource: chain as a valid S3 source, and hard-fail if
chain is combined with a non-empty s3.accessKey.

No serviceAccount change is needed — the chart already creates a dedicated SA and
exposes serviceAccount.annotations (enough for IRSA's eks.amazonaws.com/role-arn),
and Pod Identity requires nothing on the SA because the association is AWS-side.

_Workaround for anyone hitting this now_
Have your secret manager emit literal empty strings for BUZZ_S3_ACCESS_KEY and
BUZZ_S3_SECRET_KEY (e.g. an External Secrets target.template.data block rather than a
remote-ref property). The secretKeyRefs resolve to "", and the relay takes the
credential-chain branch as intended.

_Minor, related_
The aws-creds fork pin is marked temporary pending durch/rust-s3#449. Until that
lands, Pod Identity support depends on an unmerged fork — worth noting in the chart README
so operators know what their Pod Identity support actually rests on.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.