out_s3: current_buffer_size underflows after deleting buffer files recovered from a previous run — every new chunk is then rejected with "Buffer is full"
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 58
Description
## Describe the bug
The s3 output tracks store_dir disk usage in `ctx->current_buffer_size`
(a `size_t`). The accounting is asymmetric across restarts:
- buffer files left behind by a previous run (e.g. after a crash or a
pod restart with pending uploads) are adopted at startup and uploaded,
but their sizes are **never added** to `current_buffer_size`;
- when such a file is deleted after upload, `s3_store_file_delete()`
subtracts its size **unconditionally**:
```c
/* plugins/out_s3/s3_store.c */
int s3_store_file_delete(struct flb_s3 *ctx, struct s3_file *s3_file)
{
...
ctx->current_buffer_size -= s3_file->size;
```
Subtracting more than was ever added wraps the unsigned counter around
to ~2^64. From that point on, the `store_dir_limit_size` check in
`s3_store_buffer_put()` considers the buffer permanently full and
rejects **every** new chunk:
```
[error] [output:s3:out_gwlog_gcs] Buffer is full: current_buffer_size=18446744073709314048, new_data=4931, store_dir_limit_size=20000000000 bytes
[ warn] [output:s3:out_gwlog_gcs] Could not buffer chunk. Data order preservation will be compromised
[error] [engine] chunk '1-1786521063.962669295.flb' cannot be retried: task_id=1, input=re_gwlog > output=out_gwlog_gcs
```
`18446744073709314048 == 2^64 - 237568`, i.e. the counter is at
"-237568 bytes" — exactly the size of the leftover files that were
uploaded and deleted. The counter keeps drifting further "negative" as
more recovered files are deleted, and it can never recover, so the
output effectively stops accepting data until the process is restarted
(and the same thing can happen again on the next restart). This results
in permanent data loss ("chunk cannot be retried").
Note that another code path already guards against exactly this: the
quarantine path clamps the subtraction at zero. `s3_store_file_delete()`
is missing the same guard.
## To Reproduce
1. Configure an s3 output with `store_dir` on a persistent path (e.g. a
hostPath volume in Kubernetes) and `store_dir_limit_size` set (e.g.
`20G`).
2. Let Fluent Bit buffer some data, then kill it before the upload
completes, leaving files in `store_dir`.
3. Start Fluent Bit again. It adopts and uploads the leftover files.
4. As soon as the recovered files are uploaded and deleted,
`current_buffer_size` wraps to ~2^64 and every subsequent chunk is
rejected with "Buffer is full" even though the disk is empty.
## Expected behavior
Deleting a buffer file must not wrap the usage counter. Either the
subtraction should be clamped at zero (defensive fix, same pattern as
the quarantine path), or files adopted at startup should be accounted
into `current_buffer_size` (root-cause fix), or both.
## Suggested fix (minimal, defensive)
```c
if (ctx->current_buffer_size >= s3_file->size) {
ctx->current_buffer_size -= s3_file->size;
}
else {
ctx->current_buffer_size = 0;
}
```
I'm happy to submit a PR with this change.
## Workaround
Unset `store_dir_limit_size` (default 0 = unlimited): the broken check
only runs when a limit is configured.
## Your Environment
- Version used: 5.0.9 (the unguarded subtraction is also present on current master)
- Configuration: s3 output → GCS S3-compatible endpoint, `use_put_object On`, `store_dir` on a Kubernetes hostPath volume, `store_dir_limit_size 20G`
- Environment: GKE (containerd), DaemonSet tailing container logs
---
# PR Title
out_s3: guard current_buffer_size subtraction against underflow
# PR Body
Fixes #<이슈번호>
`s3_store_file_delete()` subtracted `s3_file->size` from
`ctx->current_buffer_size` unconditionally. Buffer files recovered from
a previous run are not accounted into `current_buffer_size` at startup,
so deleting them after upload wraps the unsigned counter around to
~2^64. The `store_dir_limit_size` check then treats the buffer as
permanently full and rejects every new chunk ("Buffer is full",
"chunk cannot be retried"), causing data loss until restart.
Clamp the subtraction at zero, using the same pattern already applied
on the quarantine accounting path in `s3.c`.
Observed in production on 5.0.9 (Kubernetes DaemonSet, `store_dir` on a
hostPath volume surviving pod restarts):
```
[error] Buffer is full: current_buffer_size=18446744073709314048, new_data=4931, store_dir_limit_size=20000000000 bytes
```
`18446744073709314048 == 2^64 - 237568` — the counter went "negative"
by exactly the size of the recovered-and-deleted files.
----
## Testing
- [x] Example configuration file for the change
- [x] Debug log output from testing the change
## Documentation
- [ ] N/A — bug fix, no user-facing configuration change
## Backporting
- [ ] Backport to latest stable release.
Contributor guide
Research direction
Start in plugins/out_s3/s3_store.c at s3_store_file_delete() and compare its accounting with the guarded subtraction in the quarantine path in s3.c. Reproduce the restart scenario with recovered store_dir files and a configured store_dir_limit_size, then verify deletion cannot make current_buffer_size wrap and new chunks are no longer rejected as "Buffer is full".
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100