Multiple Multiline Filter Definitions Not Supported
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 71
Description
## Bug Report
If you put two multiline filter definitions in your conf and they both match the same logs this leads to a problem:
```
[FILTER]
name multiline
match *
multiline.key_content log
multiline.parser go
[FILTER]
name multiline
match *
multiline.key_content log
multiline.parser multiline-regex-test
```
This leads to error messages like:
```
[2022/02/09 07:28:03] [error] [multiline] expected MAP type in first line state buffer
```
And can cause high memory usage and even cause Fluent Bit to crash.
### Why does this happen?
This is because the multiline filter using an emitter input instance to re-emit completed records at the start of the Fluent Bit log pipeline.
In the multiline design https://github.com/fluent/fluent-bit/issues/4309 I tried to prevent cycles by having the filter recognize its own in_emitter instance and not try to parse records from its own emitter. Unfortunately, this only solves the problem for a single filter instance. Two filters can lead to a cycle.
So let's go through an example to see why this happens:
1. Some input ingests chunk A, which contains a multiline
2. ML_FILTER_1 gets chunk A, and concatenates the records and emits them as Chunk B with the in_emitter
3. ML_FILTER_1 recognizes that Chunk B came from its own emitter, so passes it on unmodified.
4. ML_FITLER_2 gets Chunk B, processes the records (assume they match at least one parser) and then emits them as Chunk C with its in_emitter
5. ML_FILTER_1 gets chunk B, and concatenates the records (these records already passed through that filter and thus match) and emits them as Chunk C with the in_emitter. At this point, we are now at step 2 again, and we have reached an infinite loop.
### Workaround 1: If you need only one parser applied to each log statement
The workaround is to only have a single filter definition but remember that you can use multiple parsers in a single definition. The Fluent Bit multiline filtr can only apply a single multiline parser to each log record; it will try each parser in the comma delimited list in order, and apply the first one that matches the log (i.e. use the first parser which has a start_state that matches the log).
This limitation means that each log record can only have 2 multiline parsers successfully applied to it. The first appliedparser can be defined with the tail multiline settings, and the second applied parser can be specified in a multiline filter definition.
So my example from above can become a single filter definition like so:
```
[FILTER]
name multiline
match *
multiline.key_content log
multiline.parser go, multiline-regex-test
```
Workaround 2: If logs can be differentiated by log tag
Another option is if you can have the `Match` pattern for each filter match different tags.
Contributor guide
Research direction
Start by reproducing the configuration with two multiline filters matching the same logs, then trace how each filter's emitter re-enters the pipeline. The fix is complete when matching records no longer cycle between filter emitters, produce the reported error, consume excessive memory, or crash Fluent Bit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- observability, stream-processing
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100