Multiline feature with built-in CRI parser does not separate streams
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 71
Description
## Bug Report
**Describe the bug**
When a container application produces log messages that are split by the container runtime into multiple parts, and there are log messages written to stdout and stderr more or less at the same time, it is possible that Fluent Bit does not concatenate those parts.
**To Reproduce**
- Example log message:
```
2021-11-18T14:47:34.730832241+00:00 stdout F some short log message A
2021-11-18T14:47:34.730832241+00:00 stdout P some longer log message B part one
2021-11-18T14:47:34.730908438+00:00 stdout F end of the longer message B
2021-11-18T14:47:34.730908438+00:00 stdout P another longer log message C part one
2021-11-18T14:47:34.730939216+00:00 stdout P message C continued in part two
2021-11-18T14:47:34.735454549+00:00 stderr F some log message D in stderr part one
2021-11-18T14:47:34.735482854+00:00 stdout F message C still continued after a line from a different stream
2021-11-18T14:47:34.735482854+00:00 stderr F and the end of the stderr message D
```
- Steps to reproduce the problem:
Create an application that writes a couple of long lines to stdout and stderr and run in a CRI-O runtime (e.g. Openshift Kubernetes cluster).
**Expected behavior**
The output (in our case Elastic) shows each log message complete as a separate document.
**Your Environment**
* Version used: container image fluent/fluent-bit:1.8.9
* Configuration: relevant part:
```
[INPUT]
Name tail
Path /var/log/containers/*.log
Multiline.Parser cri
```
* Environment name and version: Kubernetes 1.20 / Openshift 4.7.37
* Server type and version:
* Operating System and version: RHEL 47.84, CRI-O 1.20.5-7
* Filters and plugins:
**Additional context**
* Lines have an indication in field 3: F for a one-line message and for the concluding line of a multi-line message; P for parts other than the final part of a multi-line message.
* Log messages from different streams (stdout, stderr) can be mixed up (examples C and D).
* Log messages can be in JSON and we also apply the JSON parser as filter.
* Messages which are not recombined by Fluent Bit are hard to find in Elastic and in case they are formatted as JSON, the parts can obviously not be _interpreted_ as JSON, so we are missing a lot of information in Elastic for those lines.
Note that this is not about the multiline _filter_ (see e.g. #4309), nor about the old multiline parser (see [Tail - Multiline](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline)).
Contributor guide
Assessment
This issue has not been assessed yet.