Title: Fluent Bit 5.0.7 Forward output intermittently nests complete Forward batches inside another Forward batch
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 71
Description
## Bug Report
**Describe the bug**
Fluent Bit 5.0.7 on Windows intermittently sends malformed Forward protocol messages to Fluentd 1.16.11.
Instead of placing individual `[time, record]` events inside the Forward entries array, Fluent Bit sometimes places complete `[tag, entries]` Forward batches inside another Forward batch.
Expected decoded MessagePack structure:
```text
[
"winlog",
[
[event_time, record_hash],
[event_time, record_hash]
],
options
]
```
Observed decoded structure:
```text
[
"winlog",
[
[
"winlog",
[
[event_time, record_hash],
[event_time, record_hash]
]
],
[
"winlog",
[
[event_time, record_hash],
[event_time, record_hash]
]
]
]
]
```
Fluentd interprets each nested Forward batch as an event:
```text
time = "winlog"
record = [[event_time, record_hash], ...]
```
It consequently rejects the record because its top-level type is an Array:
```text
emit transaction failed:
error_class=ArgumentError
error="record must be a Hash: Array"
tag="winlog"
```
This is followed by:
```text
unexpected error on reading data
error_class=ArgumentError
error="record must be a Hash: Array"
```
The individual Windows records inside the nested batches are valid MessagePack maps. Arrays inside record fields, such as `StringInserts`, are valid and are not the source of the error.
**To Reproduce**
- Rubular link if applicable: N/A
- Example log message if applicable:
An anonymized individual Windows record inside one of the nested batches:
```json
{
"RecordNumber": 1000001,
"TimeGenerated": "2026-07-30 13:10:01 +0300",
"TimeWritten": "2026-07-30 13:10:01 +0300",
"EventID": 4624,
"EventType": "SuccessAudit",
"Channel": "Security",
"SourceName": "Microsoft-Windows-Security-Auditing",
"ComputerName": "HOST01.example.invalid",
"Message": "",
"StringInserts": [
"",
""
]
}
```
An anonymized diagnostic entry showing how Fluentd interpreted one nested batch:
```json
{
"diagnostic_timestamp": "2026-07-30T10:20:04.133768Z",
"sender_address": "",
"incoming_tag": "winlog",
"forward_mode": "Forward",
"chunk_id": null,
"batch_size": 153,
"bad_event_index": 0,
"event_time": "\"winlog\"",
"record_class": "Array",
"array_length": 215,
"first_element_classes": [
"Array",
"Array",
"Array",
"Array"
]
}
```
The diagnostic input inspected the decoded MessagePack object immediately before native Fluentd `in_forward` processing.
- Steps to reproduce the problem:
1. Configure Fluent Bit 5.0.7 on Windows with a `winlog` input and filesystem storage.
2. Configure the Forward output to send to Fluentd 1.16.11:
```conf
[OUTPUT]
Name forward
Match winlog
Host
Port 24224
Send_options On
Require_ack_response On
Retain_Metadata_In_Forward_Mode False
Storage.Total_Limit_Size 2G
Retry_Limit False
Net.Connect_Timeout 10
Net.Keepalive On
Net.Keepalive_Idle_Timeout 30
```
3. Allow Windows Security events to accumulate in Fluent Bit’s filesystem storage.
4. Forward the stored events to Fluentd.
5. Inspect the decoded Forward messages received by Fluentd.
The problem is intermittent, and we do not yet have a deterministic minimal reproduction. It has been observed while filesystem-buffered events were being forwarded. A relationship to backlog replay, retries, acknowledgements, or connection recovery is suspected but has not been confirmed.
In one captured occurrence:
```text
Outer tag: winlog
Outer entry count: 153
Malformed indexes: 0 through 152
Nested batch count: 153
Nested event total: 34,182
Smallest nested batch: 38 events
Largest nested batch: 556 events
Average nested batch: 223.4 events
Outer chunk ID: absent
```
Every outer entry was another complete Forward batch. For every entry, Fluentd observed:
```text
event time: "winlog"
record class: Array
array members: Array, Array, Array, ...
```
**Expected behavior**
The Forward output should always serialize log entries in a standard Fluentd-compatible structure:
```text
[tag, [[time, record], [time, record], ...], options]
```
Each `record` should be the original event map.
A complete `[tag, entries]` Forward message should not appear as a record inside another Forward message.
When `Retain_Metadata_In_Forward_Mode False` is configured, the output should use strict Forward-compatible records suitable for Fluentd.
**Screenshots**
N/A.
Sanitized diagnostic files and Fluent Bit logs can be provided if helpful.
**Your Environment**
- Version used:
```text
Fluent Bit 5.0.7
Fluentd 1.16.11
```
- Configuration:
```conf
[SERVICE]
Flush 1
Daemon Off
Log_Level debug
Storage.Metrics On
Storage.Path C:\fluent-bit-data\buffer
Storage.Sync normal
Storage.Checksum On
Storage.Backlog.Mem_Limit 128M
Storage.Max_Chunks_Up 256
[INPUT]
Name winlog
Tag winlog
Channels Security
Interval_Sec 1
DB C:\fluent-bit-data\db\winlog-security.sqlite
Mem_Buf_Limit 64MB
Storage.Type filesystem
[OUTPUT]
Name forward
Match winlog
Host
Port 24224
Send_options On
Require_ack_response On
Retain_Metadata_In_Forward_Mode False
Storage.Total_Limit_Size 2G
Retry_Limit False
Net.Connect_Timeout 10
Net.Keepalive On
Net.Keepalive_Idle_Timeout 30
```
- Environment name and version:
```text
Windows host forwarding Windows Security events
```
- Server type and version:
```text
Fluentd 1.16.11 using the native in_forward implementation
```
- Operating System and version:
```text
Windows [edition and build to be added]
```
- Filters and plugins:
```text
Input: winlog
Output: forward
Storage: filesystem
Filters: [none, or list any configured filters]
```
**Additional context**
The objective is to forward Windows Security events from Fluent Bit to Fluentd and then to downstream storage without record gaps.
Most records are delivered successfully, but a malformed outer Forward batch can contain tens of thousands of otherwise-valid Windows events. Fluentd rejects the malformed stream with:
```text
record must be a Hash: Array
```
The downstream system has observed gaps in Windows `RecordNumber` values.
A temporary custom Fluentd input can recognize the exact nested shape, flatten the inner batches, and submit the recovered events to native Fluentd processing. This confirms that the individual records inside the nested batches are valid. However, this is only a receiver-side recovery measure; the unexpected Forward framing originates before Fluentd processes the message.
Could the maintainers clarify:
1. Whether nested Forward batches are intentional or supported in any Fluent Bit 5.0.7 mode.
2. Whether filesystem backlog recovery can cause already-formatted Forward messages to be wrapped a second time.
3. Whether this is a known Windows-specific Forward-output or storage issue.
4. Whether there is a supported sender-side configuration that guarantees standard Fluentd-compatible serialization.
5. What additional tracing, MessagePack capture, or filesystem-chunk information would be most useful for diagnosing this.
Contributor guide
Research direction
Start with the supplied Fluent Bit 5.0.7 Windows configuration, filesystem-buffered backlog replay, and decoded Forward messages. Compare normal output with the nested batches while investigating retries, acknowledgements, and connection recovery; done means the sender consistently emits [tag, entries, options] with individual [time, record] entries and no nested Forward batches.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- observability-sre, stream-processing
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100