fluent / fluent/fluent-bit

Built in support for Multiline JSON

Open
#8,232 2 comments 6 reactions 1 assignee Claimed by @lecaros View on GitHub
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 20h
Merged PRs (30d)
71

Description

**Is your feature request related to a problem? Please describe.**
It is very common to have JSON span multiple lines. This is an example log:
```json
{
"level": "ERROR",
"message": "This is my message",
"timestamp": "06:45:03.376"
}
```

**Describe the solution you'd like**
A built-in multiline parser similar to those for docker, cri, go etc. https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/multiline-parsing.
```
[INPUT]
name tail
path /var/log/app.log
multiline.parser json
```

**Describe alternatives you've considered**
I was unable to get it working using the new multiline core mechanism. Every solution I had resulted in a multiline JSON being sent to the JSON parser that doesn't support it. I was able to get the workarounds discussed here for the old multiline to work: https://github.com/fluent/fluent-bit/issues/2418

```
[PARSER]
Name custom_json_parser
Format json

[PARSER]
Name multiline_json
Format regex
Regex ^(?{.*)

[INPUT]
Name tail
Multiline On
Parser_Firstline multiline_json
path /var/log/app.log

[FILTER]
Name parser
Match *
Key_Name data
Parser custom_json_parser
Unescape_Key On
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.