fluent / fluent/fluent-bit

out_s3: $INDEX sequence index resets on restart because the metadata stream is deleted on shutdown (causes object overwrites)

Open
#11,987 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 16h
Merged PRs (30d)
58

Description

## Bug Report

**Fluent Bit version:** v4.2.2 (also present in v5.0.7 and current `master`)
**Plugin:** `out_s3`
**Environment:** Linux x86_64; Fluent Bit runs as a sidecar that tails rolled log files and uploads them to S3, with `store_dir` on a persistent volume that survives process/container restarts.

One caveat is that I'm unsure if this is intentional behavior or not? For context, I'm trying to mirror Spark event logs to S3 similar to [this blog post](https://aws.amazon.com/blogs/containers/inside-pinterests-custom-spark-job-logging-and-monitoring-on-amazon-eks-using-aws-for-fluent-bit-amazon-s3-and-adot-2/)

### Configuration (minimal repro)

```ini
[INPUT]
name tail
path /var/log/app/segment_*
db /data/flb/tail.db
read_from_head on

[OUTPUT]
name s3
match *
bucket my-bucket
region us-east-1
total_file_size 50M
store_dir /data/flb/s3
s3_key_format /logs/myapp/segment_$INDEX
```

`store_dir` (and the tail `db`) are on a volume that survives restarts.

### Expected behavior

Per the docs:

> `$INDEX` ... is saved in the `store_dir`. If you restart Fluent Bit with the same disk, it can continue incrementing the index from its last value in the previous run.

So after a restart on the same `store_dir`, the next upload should continue from the last value (e.g. `segment_5`), and previously uploaded objects should never be overwritten.

### Actual behavior

After a **graceful** restart (SIGTERM) with the same persistent `store_dir`, `$INDEX` resets to `0`. The next uploads recreate `segment_0`, `segment_1`, ... **overwriting** objects from the previous run. There is no `Successfully recovered index. Continuing at index=N` line on startup.

### Root cause

`$INDEX` state is stored as a **plain file** at `//sequence/index_metadata/seq_index_` (written with `fopen`/`fprintf` in `init_seq_index()` / `write_seq_index()`), inside an fstore stream named `sequence`. This file is **not** a Chunk I/O chunk, and Chunk I/O's scan only registers regular files located *directly* under a stream directory — `cio_scan_stream_files()` skips the nested `index_metadata/` subdirectory because it is not `DT_REG`. As a result the `sequence` fstore stream has **zero** registered files.

On shutdown, `cb_s3_exit()` → `s3_store_exit()` → `flb_fstore_destroy()` iterates all streams and deletes any stream with `files == 0` via `flb_fstore_stream_destroy(stream, FLB_TRUE)` → `cio_stream_delete()`, which **recursively removes the stream directory**. That deletes `sequence/index_metadata/seq_index_`. On the next start, `init_seq_index()`'s `access(seq_index_file, F_OK)` misses and resets `$INDEX` to `0`.

Relevant code (`master`):

- `plugins/out_s3/s3.c` — `init_seq_index()` stores `$INDEX` as a raw file; `access()`-or-reset logic.
- `lib/chunkio/src/cio_scan.c` — `cio_scan_stream_files()` only handles `DT_REG` entries, so the nested `index_metadata/` dir is ignored and the stream has 0 files.
- `src/flb_fstore.c` — `flb_fstore_destroy()` deletes streams with 0 files (`if (files == 0) delete = FLB_TRUE;`).
- `plugins/out_s3/s3_store.c` — `s3_store_exit()` calls `flb_fstore_destroy()`.

### Additional notes

- **Not multipart-specific:** both the PutObject and multipart upload paths depend on the same `init_seq_index()` state file.
- **Confirmed with a marker + inode test:** a marker file placed *inside* `sequence/` is gone after a graceful restart, while a marker placed one level *above* it (still on the same persistent volume) survives — i.e. the volume is healthy; Fluent Bit deletes its own metadata stream. The `sequence/` directory comes back with a fresh inode and mtime after the restart.
- A hard crash (SIGKILL) does **not** delete it, because `cb_s3_exit()` / `flb_fstore_destroy()` only run on graceful shutdown — but typical restarts use SIGTERM, which does.

### Proposed fix

Detach the metadata stream reference in `s3_store_exit()` before `flb_fstore_destroy()` runs, so the empty-stream cleanup skips it and the persisted `$INDEX` file survives:

```c
if (ctx->stream_metadata != NULL) {
flb_fstore_stream_destroy(ctx->stream_metadata, FLB_FALSE);
ctx->stream_metadata = NULL;
}
```

I have a fix ready and will open a PR shortly.

Contributor guide

Open the contributing guide

Research direction

Start with plugins/out_s3/s3.c and trace init_seq_index() alongside plugins/out_s3/s3_store.c during graceful shutdown. Read lib/chunkio/src/cio_scan.c and src/flb_fstore.c to confirm how the nested metadata directory becomes an empty stream and is removed. Reproduce a SIGTERM restart with a persistent store_dir; done means the next run recovers the prior index without overwriting objects.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, c
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.