fluent / fluent/fluent-bit

in_tail & out_s3: Support for continuously growing files with strict ordering, sequential delivery, and atomic completeness

Open
#12,093 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 16h
Merged PRs (30d)
58

Description

**Is your feature request related to a problem? Please describe.**

It would be helpful if the S3 output plugin could better support continuously growing (append-only) files when uploading to object storage.

One common use case is collecting continuously growing log files. Depending on the application and log rotation policy, a log file may remain open and grow for an extended period, or new log files may be created periodically as existing files reach a configured size or age. In either case, log data is generated continuously, and the S3 output plugin may trigger uploads due to upload_timeout or total_file_size before log generation has completed. Periodic uploads allow the latest logs to be available in object storage without waiting for the application or logging process to finish.

**Background**

In our deployment, Fluent Bit tails event log files and uploads them to an S3 compatible object store.

The log collection flow is:
- Tail input plugin monitors the Spark event log and forwards newly appended data to the Fluent Bit engine.
- The engine buffers the records.
- The S3 output plugin uploads buffered data when either upload_timeout or total_file_size is reached.

When successive uploads target the same object key (for example, using static_file_path On with a constant object key), each upload replaces the previously stored object with only the data contained in the current upload.

As a result, earlier portions of the log are no longer present in the destination object.

**Example**

At time T1, the event log contains:
```
event1
event2
event3
```
upload_timeout expires, and Fluent Bit uploads:
```
event1
event2
event3
```
The application continues running and appends additional events:
```
event1
event2
event3
event4
event5
```
At the next upload cycle, Fluent Bit uploads only the newly buffered records:
```
event4
event5
```

Since the destination object key is unchanged, the object stored in object storage now contains only:
```
event4
event5
```

The previously uploaded events are no longer part of the object. **Changing the object key breaks the application that reads the uploaded content.**

**Use case**
This behavior is particularly relevant for append-only log files that remain open for long periods, such as:
- Spark event logs
- Database audit logs
- Long-running application logs
- Other continuously growing log files

For these workloads, users may want both:
- Near real-time availability of logs in object storage.
- A single object that always represents the complete log collected so far.

**Current workaround**
At the moment, the available options appear to be:
- Configure a large upload_timeout so uploads occur only after the application has mostly finished, reducing the chance of partial objects but delaying log availability.
- Configure a small upload_timeout to make logs available sooner, but the destination object only reflects the most recently uploaded portion of the log.

Both approaches involve trade-offs depending on the workload.

**Describe the solution you'd like**
Would it be possible to consider a mode that better supports continuously growing files?

Some possible approaches could include:

- Re-uploading the complete file contents when a static object path is used.
- Providing an optional mode that maintains a complete object for append-only source files.
- Supporting backend-specific append semantics where they are available.

One approach I considered:
1. Tail plugin reading whole file when a certain flag is set (Flag is used to avoid changing default behaviour)
2. S3 plugin receives whole file after clearing existing buffer

This way, the whole file is written to target storage every time. **Trade-off:** A higher CPU / IO usage from fluent bit container / process.

These are only suggestions - the maintainers may have better approaches that fit the existing architecture.

**Describe alternatives you've considered**
-

**Additional context**
-

Contributor guide

Open the contributing guide

Research direction

Start by tracing the in_tail input and out_s3 output plugin entry points, focusing on upload_timeout, total_file_size, and static_file_path handling. Compare how successive uploads are buffered and written, then determine how strict ordering, sequential delivery, and atomic completeness could be specified. Done should preserve earlier records in the configured object while retaining current upload behavior by default.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, c
Domain
backend, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.