fluent / fluent/fluent-bit

How to read multiline json format

Open
#2,418 7 comments 0 reactions 1 assignee Claimed by @lecaros View on GitHub
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 16h
Merged PRs (30d)
58

Description

I'm trying to read below json format but reading line by line .

td-agent-bit version is 1.4.5

**Log format** :-

{
"thread" : "1",
"level" : "OFF",
"loggerName" : "AuditLogger",
"marker" : {
"name" : "Audit",
"parents" : [ {
"name" : "EVENT"
} ]
},
"message" : "Audit [useCredentials fileName=\"Credential id=apitoken name=paasjenkins/****** (apitoken-nonprod-jobs)\" name=\"Credential id=apitoken name=paasjenkins/****** (apitoken-nonprod-jobs)\" timestamp=\"2020-06-02T12:34:36.928+04:00\" usage=\"[11,1864)\"]",
"endOfBatch" : false,
"loggerFqcn" : "org.apache.logging.log4j.audit.AuditLogger",
"instant" : {
"epochSecond" : 1594986664,
"nanoOfSecond" : 204000000
},
"contextMap" : { },
"threadId" : 129467,
"threadPriority" : 5
}
{
"thread" : "2",
"level" : "OFF",
"loggerName" : "AuditLogger",
"marker" : {
"name" : "Audit",
"parents" : [ {
"name" : "EVENT"
} ]
},
"message" : "Audit [useCredentials fileName=\"Credential id=apitoken name=paasjenkins/****** (apitoken-nonprod-jobs)\" name=\"Credential id=apitoken name=paasjenkins/****** (apitoken-nonprod-jobs)\" timestamp=\"2020-06-02T12:34:36.928+04:00\" usage=\"[11,1864)\"]",
"endOfBatch" : false,
"loggerFqcn" : "org.apache.logging.log4j.audit.AuditLogger",
"instant" : {
"epochSecond" : 1594986664,
"nanoOfSecond" : 204000000
},
"contextMap" : { },
"threadId" : 129467,
"threadPriority" : 5
}

**td-agent-bit.conf**
[SERVICE]
Parsers_File /etc/td-agent-bit/parsers.conf
Daemon off
Flush 5
Log_Level trace

[INPUT]
Name tail
Path /var/log/jenkins/audit1.log
DB /etc/td-agent-bit/db/enbd-jenkins-audit.db
Path_Key tailed.path
Tag enbd-jenkins-audit
Parser_Firstline firstline2
Multiline On
Parser_1 multiline2
Refresh_Interval 5
Buffer_Chunk_Size 3M
Buffer_Max_Size 200M
Mem_Buf_Limit 300M
Skip_Long_Lines true
[OUTPUT]
Name stdout
Match enbd-jenkins-*

**parsers.conf**
[PARSER]
Name firstline2
Format regex
Regex ^(?\{\n\s{1,}\"thread\"\s:)

[PARSER]
Name multiline2
Format regex
Regex (?m-ix)^(?\{\n\s{1,}\"thread\".*)

I want to read whole thread-1 is one message and other one is another message.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.