aws credentials: use-after-free when refresh races get_credentials() leading to SIGSEGV with multi-worker outputs (EKS/IRSA)
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 71
Description
## Bug Report
**Describe the bug**
Fluent Bit crashes with SIGSEGV under normal operation when an AWS output plugin is configured with `workers > 1` and uses the EKS (IRSA) or STS credential provider. The cause is a use-after-free race in the credential providers: `get_credentials_fn_eks()` copies the cached credentials **without holding the provider lock**, while the refresh path frees and replaces those credentials **under the lock**. The code is identical from at least v1.8 through v5.0.9 and current master, so all releases are affected. Across our EKS fleet (~12,000 DaemonSet pods in two regions), this hits about 500 unique times a day.
**The race**
The writer, [`sts_assume_role_request()` L776-L781](https://github.com/fluent/fluent-bit/blob/v4.2.0/src/aws/flb_aws_credentials_sts.c#L776-L781), runs with the provider lock held (via `try_lock_provider()` in [`refresh_fn_eks()` L478-L488](https://github.com/fluent/fluent-bit/blob/v4.2.0/src/aws/flb_aws_credentials_sts.c#L478-L488)):
```c
/* unset and free existing credentials first */
flb_aws_credentials_destroy(*creds); /* frees the struct readers may hold */
*creds = NULL;
*next_refresh = expiration - FLB_AWS_REFRESH_WINDOW;
*creds = credentials;
```
The reader, [`get_credentials_fn_eks()` L406-L476](https://github.com/fluent/fluent-bit/blob/v4.2.0/src/aws/flb_aws_credentials_sts.c#L406-L476), only takes the lock on the refresh branch. When credentials are cached and not due for refresh — the common case, every flush — it reads them with **no lock at all**:
```c
if (!implementation->creds || refresh == FLB_TRUE) {
if (try_lock_provider(provider)) {
assume_with_web_identity(implementation);
unlock_provider(provider);
}
}
if (!implementation->creds) { /* L430 — unlocked NULL check */
return NULL;
}
creds = flb_calloc(1, sizeof(struct flb_aws_credentials));
/* ... */
creds->access_key_id = flb_sds_create(implementation->creds->access_key_id); /* L447 — unlocked deref */
creds->secret_access_key = flb_sds_create(implementation->creds->secret_access_key);
/* ... */
```
Interleaving: worker A passes the NULL check at L430; worker B (time-based refresh, or the auth-error handler at [`flb_aws_util.c` L199-L207](https://github.com/fluent/fluent-bit/blob/v4.2.0/src/aws/flb_aws_util.c#L199-L207), [L229-L237 at v5.0.9](https://github.com/fluent/fluent-bit/blob/v5.0.9/src/aws/flb_aws_util.c#L229-L237)) completes a refresh and runs `flb_aws_credentials_destroy(*creds)`; worker A then dereferences the stale pointer at L447. Depending on timing this produces:
1. a read of freed memory → SIGSEGV, or a garbage signature (`[signv4] error hashing canonical request` / `InvalidSignatureException`);
2. a NULL dereference, if A lands between `*creds = NULL` and the re-assignment;
3. a silently mismatched key pair (e.g. old `access_key_id` copied before the swap, new `secret_access_key` after) → `InvalidSignatureException` with no crash.
This also happens with no auth errors at all: the time-based refresh path has the same shape. The worker that wins `try_lock_provider()` refreshes while the other N−1 workers fall through to the unlocked copy.
The comment on the provider struct ([`flb_aws_credentials.h` L119-L127](https://github.com/fluent/fluent-bit/blob/v4.2.0/include/fluent-bit/flb_aws_credentials.h#L119-L127)) shows the lock was added for exactly this hazard ("When one thread refreshes, the cached creds are freed and reset, there could be a double free without a lock"), but only the refresh side takes it. The unlocked read was safe when flushes were coroutines on one thread; `workers N` runs flushes on N real pthreads (confirmed via `/proc/1/status` Threads count), making it a genuine data race.
**Same pattern in other providers:** [`get_credentials_fn_sts()` L97-L171 (copy at L143)](https://github.com/fluent/fluent-bit/blob/v4.2.0/src/aws/flb_aws_credentials_sts.c#L143), [`get_credentials_fn_ec2()`](https://github.com/fluent/fluent-bit/blob/v4.2.0/src/aws/flb_aws_credentials_ec2.c#L97), [`get_credentials_fn_http()`](https://github.com/fluent/fluent-bit/blob/v4.2.0/src/aws/flb_aws_credentials_http.c#L123), and [`get_credentials_fn_profile()`](https://github.com/fluent/fluent-bit/blob/v4.2.0/src/aws/flb_aws_credentials_profile.c#L126) all do the same unlocked copy, so ECS, EC2-instance-role, and profile deployments are exposed too, not just EKS/IRSA.
**To Reproduce**
The race window in an unmodified binary is nanoseconds, so we built v4.2.0 with a single 10 ms `usleep()` between the destroy and the NULL assignment to make it deterministic. This changes no logic — it only widens the existing window:
```c
/* flb_aws_credentials_sts.c, sts_assume_role_request() */
flb_aws_credentials_destroy(*creds);
usleep(10000); /* widen the race window from ns to 10ms */
*creds = NULL;
```
Deployed on EKS: 14 replicas, `cloudwatch_logs` output with `workers 25`, `Flush 0.1`, IRSA credentials. To force a refresh on demand we attached an IAM deny on `logs:PutLogEvents`, which produces `AccessDeniedException` → the auth-error handler calls `refresh(provider)` → destroy/replace runs while workers are signing.
**Result: 8 of 14 pods crashed with exit 139 within 30 seconds.** Stack trace from a crashed pod:
```
#0 0x563db6bca383 flb_sds_create() at src/flb_sds.c:86
#1 0x563db706d8b3 get_credentials_fn_eks() at src/aws/flb_aws_credentials_sts.c:448
#2 0x563db6c3f3d9 flb_signv4_do() at src/flb_signv4.c:1163
#3 0x563db7068e0e request_do() at src/aws/flb_aws_util.c:503
#4 0x563db7069a2d flb_aws_client_request() at src/aws/flb_aws_util.c:182
#5 0x563db6d36dc1 put_log_events() at plugins/out_cloudwatch_logs/cloudwatch_api.c:2035
#6 0x563db6d379c3 send_log_events() at plugins/out_cloudwatch_logs/cloudwatch_api.c:816
#8 0x563db6d38817 process_and_send() at plugins/out_cloudwatch_logs/cloudwatch_api.c:1577
#9 0x563db6d317ce cb_cloudwatch_flush() at plugins/out_cloudwatch_logs/cloudwatch_logs.c:457
#10 0x563db72a4d8a co_init() at lib/monkey/deps/flb_libco/amd64.c:117
```
Frame #1 reports line 448 because the repro patch adds one `#include` line at the top of the file; it is line 447 at the `v4.2.0` tag — the `flb_sds_create(implementation->creds->access_key_id)` copy shown above.
Repro Dockerfile (multi-stage, applies the usleep patch via sed)
```dockerfile
FROM public.ecr.aws/docker/library/ubuntu:22.04 AS builder
RUN apt-get update && apt-get install -y \
git cmake build-essential flex bison pkg-config \
libssl-dev libyaml-dev libsystemd-dev libpq-dev \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /src
RUN git clone --depth 1 --branch v4.2.0 https://github.com/fluent/fluent-bit.git
WORKDIR /src/fluent-bit
RUN sed -i '1i #include ' src/aws/flb_aws_credentials_sts.c && \
sed -i 's/flb_aws_credentials_destroy(\*creds);/flb_aws_credentials_destroy(*creds); usleep(10000);/' \
src/aws/flb_aws_credentials_sts.c
RUN cmake -B build -DCMAKE_BUILD_TYPE=Release -DFLB_RELEASE=On -DFLB_DEBUG=Off \
-DFLB_EXAMPLES=Off -DFLB_SHARED_LIB=Off \
&& cmake --build build -j$(nproc)
FROM public.ecr.aws/docker/library/ubuntu:22.04
RUN apt-get update && apt-get install -y --no-install-recommends \
libssl3 libyaml-0-2 libsystemd0 libpq5 ca-certificates \
&& rm -rf /var/lib/apt/lists/*
COPY --from=builder /src/fluent-bit/build/bin/fluent-bit /fluent-bit/bin/fluent-bit
COPY --from=builder /src/fluent-bit/conf/parsers.conf /fluent-bit/parsers/parsers.conf
RUN mkdir -p /fluent-bit/etc
ENTRYPOINT ["/fluent-bit/bin/fluent-bit"]
```
Repro fluent-bit.conf
```ini
[SERVICE]
Flush 0.1
Grace 1
Daemon Off
Log_Level info
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_PORT 2021
storage.path /var/log/flb-repro-storage/buffer/
storage.sync normal
storage.metrics on
Parsers_File /fluent-bit/parsers/parsers.conf
[INPUT]
Name tail
Tag segv.*
Path /var/log/flb-repro-input/*.log
Parser json
DB /var/log/flb-repro-storage/tail.db
Mem_Buf_Limit 20MB
Skip_Long_Lines On
Refresh_Interval 1
storage.type filesystem
[OUTPUT]
Name cloudwatch_logs
Match segv.*
region us-west-2
log_group_name /aws/eks/fluentbit-segv-repro/uaf-test
log_stream_prefix segv-
auto_create_group true
workers 25
retry_limit 10
storage.total_limit_size 256M
net.keepalive On
net.dns.mode TCP
```
**Expected behavior**
`get_credentials` returns a self-consistent copy of the cached credentials, or NULL if a refresh is in flight — never a copy of freed memory, and never a crash.
**Your Environment**
* Version used: Fluent Bit 4.2.0 (shipped in aws-for-fluent-bit 3.1.1); `src/aws/flb_aws_credentials_sts.c` verified byte-identical at v5.0.9 and current master, and the same pattern is present as far back as v1.8.x
* Configuration: `tail` input → `cloudwatch_logs` output, `workers 8` in production (25 in the repro), `Flush 1` (0.1 in the repro)
* Environment: Amazon EKS, DaemonSet, credentials via IRSA (`AWS_WEB_IDENTITY_TOKEN_FILE`), service-account token expiry 900 s, STS session duration 3600 s (default)
* Server type and version: EC2 worker nodes
* Filters and plugins: in_tail, out_cloudwatch_logs (production also runs out_s3)
Contributor guide
Research direction
Start with src/aws/flb_aws_credentials_sts.c, especially get_credentials_fn_eks(), get_credentials_fn_sts(), refresh_fn_eks(), and sts_assume_role_request(); then inspect the analogous readers in flb_aws_credentials_ec2.c, flb_aws_credentials_http.c, and flb_aws_credentials_profile.c. Build the repro from the issue's Dockerfile and verify that concurrent credential reads return a consistent copy or NULL during refresh without SIGSEGV or signature errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, c
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100