fluent / fluent/fluent-bit

aws credentials: use-after-free when refresh races get_credentials() leading to SIGSEGV with multi-worker outputs (EKS/IRSA)

Open
#12,206 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 20h
Merged PRs (30d)
71

Description

## Bug Report

**Describe the bug**

Fluent Bit crashes with SIGSEGV under normal operation when an AWS output plugin is configured with `workers > 1` and uses the EKS (IRSA) or STS credential provider. The cause is a use-after-free race in the credential providers: `get_credentials_fn_eks()` copies the cached credentials **without holding the provider lock**, while the refresh path frees and replaces those credentials **under the lock**. The code is identical from at least v1.8 through v5.0.9 and current master, so all releases are affected. Across our EKS fleet (~12,000 DaemonSet pods in two regions), this hits about 500 unique times a day.

**The race**

The writer, [`sts_assume_role_request()` L776-L781](https://github.com/fluent/fluent-bit/blob/v4.2.0/src/aws/flb_aws_credentials_sts.c#L776-L781), runs with the provider lock held (via `try_lock_provider()` in [`refresh_fn_eks()` L478-L488](https://github.com/fluent/fluent-bit/blob/v4.2.0/src/aws/flb_aws_credentials_sts.c#L478-L488)):

```c
/* unset and free existing credentials first */
flb_aws_credentials_destroy(*creds); /* frees the struct readers may hold */
*creds = NULL;

*next_refresh = expiration - FLB_AWS_REFRESH_WINDOW;
*creds = credentials;
```

The reader, [`get_credentials_fn_eks()` L406-L476](https://github.com/fluent/fluent-bit/blob/v4.2.0/src/aws/flb_aws_credentials_sts.c#L406-L476), only takes the lock on the refresh branch. When credentials are cached and not due for refresh — the common case, every flush — it reads them with **no lock at all**:

```c
if (!implementation->creds || refresh == FLB_TRUE) {
if (try_lock_provider(provider)) {
assume_with_web_identity(implementation);
unlock_provider(provider);
}
}

if (!implementation->creds) { /* L430 — unlocked NULL check */
return NULL;
}

creds = flb_calloc(1, sizeof(struct flb_aws_credentials));
/* ... */
creds->access_key_id = flb_sds_create(implementation->creds->access_key_id); /* L447 — unlocked deref */
creds->secret_access_key = flb_sds_create(implementation->creds->secret_access_key);
/* ... */
```

Interleaving: worker A passes the NULL check at L430; worker B (time-based refresh, or the auth-error handler at [`flb_aws_util.c` L199-L207](https://github.com/fluent/fluent-bit/blob/v4.2.0/src/aws/flb_aws_util.c#L199-L207), [L229-L237 at v5.0.9](https://github.com/fluent/fluent-bit/blob/v5.0.9/src/aws/flb_aws_util.c#L229-L237)) completes a refresh and runs `flb_aws_credentials_destroy(*creds)`; worker A then dereferences the stale pointer at L447. Depending on timing this produces:

1. a read of freed memory → SIGSEGV, or a garbage signature (`[signv4] error hashing canonical request` / `InvalidSignatureException`);
2. a NULL dereference, if A lands between `*creds = NULL` and the re-assignment;
3. a silently mismatched key pair (e.g. old `access_key_id` copied before the swap, new `secret_access_key` after) → `InvalidSignatureException` with no crash.

This also happens with no auth errors at all: the time-based refresh path has the same shape. The worker that wins `try_lock_provider()` refreshes while the other N−1 workers fall through to the unlocked copy.

The comment on the provider struct ([`flb_aws_credentials.h` L119-L127](https://github.com/fluent/fluent-bit/blob/v4.2.0/include/fluent-bit/flb_aws_credentials.h#L119-L127)) shows the lock was added for exactly this hazard ("When one thread refreshes, the cached creds are freed and reset, there could be a double free without a lock"), but only the refresh side takes it. The unlocked read was safe when flushes were coroutines on one thread; `workers N` runs flushes on N real pthreads (confirmed via `/proc/1/status` Threads count), making it a genuine data race.

**Same pattern in other providers:** [`get_credentials_fn_sts()` L97-L171 (copy at L143)](https://github.com/fluent/fluent-bit/blob/v4.2.0/src/aws/flb_aws_credentials_sts.c#L143), [`get_credentials_fn_ec2()`](https://github.com/fluent/fluent-bit/blob/v4.2.0/src/aws/flb_aws_credentials_ec2.c#L97), [`get_credentials_fn_http()`](https://github.com/fluent/fluent-bit/blob/v4.2.0/src/aws/flb_aws_credentials_http.c#L123), and [`get_credentials_fn_profile()`](https://github.com/fluent/fluent-bit/blob/v4.2.0/src/aws/flb_aws_credentials_profile.c#L126) all do the same unlocked copy, so ECS, EC2-instance-role, and profile deployments are exposed too, not just EKS/IRSA.

**To Reproduce**

The race window in an unmodified binary is nanoseconds, so we built v4.2.0 with a single 10 ms `usleep()` between the destroy and the NULL assignment to make it deterministic. This changes no logic — it only widens the existing window:

```c
/* flb_aws_credentials_sts.c, sts_assume_role_request() */
flb_aws_credentials_destroy(*creds);
usleep(10000); /* widen the race window from ns to 10ms */
*creds = NULL;
```

Deployed on EKS: 14 replicas, `cloudwatch_logs` output with `workers 25`, `Flush 0.1`, IRSA credentials. To force a refresh on demand we attached an IAM deny on `logs:PutLogEvents`, which produces `AccessDeniedException` → the auth-error handler calls `refresh(provider)` → destroy/replace runs while workers are signing.

**Result: 8 of 14 pods crashed with exit 139 within 30 seconds.** Stack trace from a crashed pod:

```
#0 0x563db6bca383 flb_sds_create() at src/flb_sds.c:86
#1 0x563db706d8b3 get_credentials_fn_eks() at src/aws/flb_aws_credentials_sts.c:448
#2 0x563db6c3f3d9 flb_signv4_do() at src/flb_signv4.c:1163
#3 0x563db7068e0e request_do() at src/aws/flb_aws_util.c:503
#4 0x563db7069a2d flb_aws_client_request() at src/aws/flb_aws_util.c:182
#5 0x563db6d36dc1 put_log_events() at plugins/out_cloudwatch_logs/cloudwatch_api.c:2035
#6 0x563db6d379c3 send_log_events() at plugins/out_cloudwatch_logs/cloudwatch_api.c:816
#8 0x563db6d38817 process_and_send() at plugins/out_cloudwatch_logs/cloudwatch_api.c:1577
#9 0x563db6d317ce cb_cloudwatch_flush() at plugins/out_cloudwatch_logs/cloudwatch_logs.c:457
#10 0x563db72a4d8a co_init() at lib/monkey/deps/flb_libco/amd64.c:117
```

Frame #1 reports line 448 because the repro patch adds one `#include` line at the top of the file; it is line 447 at the `v4.2.0` tag — the `flb_sds_create(implementation->creds->access_key_id)` copy shown above.

Repro Dockerfile (multi-stage, applies the usleep patch via sed)

```dockerfile
FROM public.ecr.aws/docker/library/ubuntu:22.04 AS builder
RUN apt-get update && apt-get install -y \
git cmake build-essential flex bison pkg-config \
libssl-dev libyaml-dev libsystemd-dev libpq-dev \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /src
RUN git clone --depth 1 --branch v4.2.0 https://github.com/fluent/fluent-bit.git
WORKDIR /src/fluent-bit
RUN sed -i '1i #include ' src/aws/flb_aws_credentials_sts.c && \
sed -i 's/flb_aws_credentials_destroy(\*creds);/flb_aws_credentials_destroy(*creds); usleep(10000);/' \
src/aws/flb_aws_credentials_sts.c
RUN cmake -B build -DCMAKE_BUILD_TYPE=Release -DFLB_RELEASE=On -DFLB_DEBUG=Off \
-DFLB_EXAMPLES=Off -DFLB_SHARED_LIB=Off \
&& cmake --build build -j$(nproc)

FROM public.ecr.aws/docker/library/ubuntu:22.04
RUN apt-get update && apt-get install -y --no-install-recommends \
libssl3 libyaml-0-2 libsystemd0 libpq5 ca-certificates \
&& rm -rf /var/lib/apt/lists/*
COPY --from=builder /src/fluent-bit/build/bin/fluent-bit /fluent-bit/bin/fluent-bit
COPY --from=builder /src/fluent-bit/conf/parsers.conf /fluent-bit/parsers/parsers.conf
RUN mkdir -p /fluent-bit/etc
ENTRYPOINT ["/fluent-bit/bin/fluent-bit"]
```

Repro fluent-bit.conf

```ini
[SERVICE]
Flush 0.1
Grace 1
Daemon Off
Log_Level info
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_PORT 2021
storage.path /var/log/flb-repro-storage/buffer/
storage.sync normal
storage.metrics on
Parsers_File /fluent-bit/parsers/parsers.conf

[INPUT]
Name tail
Tag segv.*
Path /var/log/flb-repro-input/*.log
Parser json
DB /var/log/flb-repro-storage/tail.db
Mem_Buf_Limit 20MB
Skip_Long_Lines On
Refresh_Interval 1
storage.type filesystem

[OUTPUT]
Name cloudwatch_logs
Match segv.*
region us-west-2
log_group_name /aws/eks/fluentbit-segv-repro/uaf-test
log_stream_prefix segv-
auto_create_group true
workers 25
retry_limit 10
storage.total_limit_size 256M
net.keepalive On
net.dns.mode TCP
```

**Expected behavior**

`get_credentials` returns a self-consistent copy of the cached credentials, or NULL if a refresh is in flight — never a copy of freed memory, and never a crash.

**Your Environment**

* Version used: Fluent Bit 4.2.0 (shipped in aws-for-fluent-bit 3.1.1); `src/aws/flb_aws_credentials_sts.c` verified byte-identical at v5.0.9 and current master, and the same pattern is present as far back as v1.8.x
* Configuration: `tail` input → `cloudwatch_logs` output, `workers 8` in production (25 in the repro), `Flush 1` (0.1 in the repro)
* Environment: Amazon EKS, DaemonSet, credentials via IRSA (`AWS_WEB_IDENTITY_TOKEN_FILE`), service-account token expiry 900 s, STS session duration 3600 s (default)
* Server type and version: EC2 worker nodes
* Filters and plugins: in_tail, out_cloudwatch_logs (production also runs out_s3)

Contributor guide

Open the contributing guide

Research direction

Start with src/aws/flb_aws_credentials_sts.c, especially get_credentials_fn_eks(), get_credentials_fn_sts(), refresh_fn_eks(), and sts_assume_role_request(); then inspect the analogous readers in flb_aws_credentials_ec2.c, flb_aws_credentials_http.c, and flb_aws_credentials_profile.c. Build the repro from the issue's Dockerfile and verify that concurrent credential reads return a consistent copy or NULL during refresh without SIGSEGV or signature errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, c
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.