fluent / fluent/fluent-bit

out_s3: current_buffer_size underflows after deleting buffer files recovered from a previous run — every new chunk is then rejected with "Buffer is full"

Open Beginner friendly
#12,270 0 comments 0 reactions 0 assignees View on GitHub
status: waiting-for-triage
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 16h
Merged PRs (30d)
58

Description

## Describe the bug

The s3 output tracks store_dir disk usage in `ctx->current_buffer_size`
(a `size_t`). The accounting is asymmetric across restarts:

- buffer files left behind by a previous run (e.g. after a crash or a
pod restart with pending uploads) are adopted at startup and uploaded,
but their sizes are **never added** to `current_buffer_size`;
- when such a file is deleted after upload, `s3_store_file_delete()`
subtracts its size **unconditionally**:

```c
/* plugins/out_s3/s3_store.c */
int s3_store_file_delete(struct flb_s3 *ctx, struct s3_file *s3_file)
{
...
ctx->current_buffer_size -= s3_file->size;
```

Subtracting more than was ever added wraps the unsigned counter around
to ~2^64. From that point on, the `store_dir_limit_size` check in
`s3_store_buffer_put()` considers the buffer permanently full and
rejects **every** new chunk:

```
[error] [output:s3:out_gwlog_gcs] Buffer is full: current_buffer_size=18446744073709314048, new_data=4931, store_dir_limit_size=20000000000 bytes
[ warn] [output:s3:out_gwlog_gcs] Could not buffer chunk. Data order preservation will be compromised
[error] [engine] chunk '1-1786521063.962669295.flb' cannot be retried: task_id=1, input=re_gwlog > output=out_gwlog_gcs
```

`18446744073709314048 == 2^64 - 237568`, i.e. the counter is at
"-237568 bytes" — exactly the size of the leftover files that were
uploaded and deleted. The counter keeps drifting further "negative" as
more recovered files are deleted, and it can never recover, so the
output effectively stops accepting data until the process is restarted
(and the same thing can happen again on the next restart). This results
in permanent data loss ("chunk cannot be retried").

Note that another code path already guards against exactly this: the
quarantine path clamps the subtraction at zero. `s3_store_file_delete()`
is missing the same guard.

## To Reproduce

1. Configure an s3 output with `store_dir` on a persistent path (e.g. a
hostPath volume in Kubernetes) and `store_dir_limit_size` set (e.g.
`20G`).
2. Let Fluent Bit buffer some data, then kill it before the upload
completes, leaving files in `store_dir`.
3. Start Fluent Bit again. It adopts and uploads the leftover files.
4. As soon as the recovered files are uploaded and deleted,
`current_buffer_size` wraps to ~2^64 and every subsequent chunk is
rejected with "Buffer is full" even though the disk is empty.

## Expected behavior

Deleting a buffer file must not wrap the usage counter. Either the
subtraction should be clamped at zero (defensive fix, same pattern as
the quarantine path), or files adopted at startup should be accounted
into `current_buffer_size` (root-cause fix), or both.

## Suggested fix (minimal, defensive)

```c
if (ctx->current_buffer_size >= s3_file->size) {
ctx->current_buffer_size -= s3_file->size;
}
else {
ctx->current_buffer_size = 0;
}
```

I'm happy to submit a PR with this change.

## Workaround

Unset `store_dir_limit_size` (default 0 = unlimited): the broken check
only runs when a limit is configured.

## Your Environment

- Version used: 5.0.9 (the unguarded subtraction is also present on current master)
- Configuration: s3 output → GCS S3-compatible endpoint, `use_put_object On`, `store_dir` on a Kubernetes hostPath volume, `store_dir_limit_size 20G`
- Environment: GKE (containerd), DaemonSet tailing container logs

---

# PR Title

out_s3: guard current_buffer_size subtraction against underflow

# PR Body

Fixes #<이슈번호>

`s3_store_file_delete()` subtracted `s3_file->size` from
`ctx->current_buffer_size` unconditionally. Buffer files recovered from
a previous run are not accounted into `current_buffer_size` at startup,
so deleting them after upload wraps the unsigned counter around to
~2^64. The `store_dir_limit_size` check then treats the buffer as
permanently full and rejects every new chunk ("Buffer is full",
"chunk cannot be retried"), causing data loss until restart.

Clamp the subtraction at zero, using the same pattern already applied
on the quarantine accounting path in `s3.c`.

Observed in production on 5.0.9 (Kubernetes DaemonSet, `store_dir` on a
hostPath volume surviving pod restarts):

```
[error] Buffer is full: current_buffer_size=18446744073709314048, new_data=4931, store_dir_limit_size=20000000000 bytes
```

`18446744073709314048 == 2^64 - 237568` — the counter went "negative"
by exactly the size of the recovered-and-deleted files.

----

## Testing
- [x] Example configuration file for the change
- [x] Debug log output from testing the change

## Documentation
- [ ] N/A — bug fix, no user-facing configuration change

## Backporting
- [ ] Backport to latest stable release.

Contributor guide

Open the contributing guide

Research direction

Start in plugins/out_s3/s3_store.c at s3_store_file_delete() and compare its accounting with the guarded subtraction in the quarantine path in s3.c. Reproduce the restart scenario with recovered store_dir files and a configured store_dir_limit_size, then verify deletion cannot make current_buffer_size wrap and new chunks are no longer rejected as "Buffer is full".

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.