google / google/flogger

Mapping MetadataKey and Tags to separate log objects fields and out of the log statement/message

Open
#309 4 comments 2 reactions 2 assignees Claimed by @hagbard View on GitHub
P3 type=addition
Dominant language
Java
Stars
1.5k
Forks
133
Avg merge
6m
Merged PRs (30d)
7

Description

Hi Floggers :)

I have some questions in relation to extending Flogger, while using Logback as the logging backend, and the resulting JSON log statement output.
This seemed like the right place since the Google Group does appear to be equally active.

I have looked through the issues here and I think my question somewhat overlaps with :
* [#145 Log4j2 - Request support for ThreadContext (MDC)](https://github.com/google/flogger/issues/145)
* [#275 Support change the context prefix and suffix format of KeyValueFormatter](https://github.com/google/flogger/issues/275)

Question about log output format:

I have implemented an extension of the AbstractLogger which is working as it should - it has log APIs specific to my domain which is one of main drivers for using Flogger.
However, when I am logging in JSON format, I am getting log statements similar to this:

```json
{
"@timestamp":"2022-01-24T08:25:27.303+01:00",
"@version":"1",
"message":"This is the log statement with the argument value: '42' end of log statement.
[CONTEXT mySingleMetaKeyA=\"valueA\" myrepeatedKeyB=\"valueX\" myrepeatedKeyB=\"valueZ\" myTimestampMS=1643009127300 ]",
"logger_name":"mypackage.SpringBootApplication",
"thread_name":"restartedMain","level":"ERROR","level_value":40000
}
```

Instead I would like to get the following JSON output for the log object:
```json
{
"@timestamp":"2022-01-24T08:25:27.303+01:00",
"@version":"1",
"message":"This is the log statement with the argument value: '42' end of log statement." ,
"logger_name":"mypackage.SpringBootApplication",
"thread_name":"restartedMain","level":"ERROR","level_value":40000,
"metadata":{
"mySingleMetaKeyA": "valueA",
"myRepeatedKeyB":["valueX", "valueZ"],
"myTimestampMS":1643009127300
}
}
```

My goal is to have the metadata key/values and tags, which I have used in my custom log API with the `with()` method, extracted into either a `metadata` field containing the actual value pairs as shown above or to simply have all the entries placed in the root level of the log object.

The motivation being that this log statement is to be picked up by an Elastic Filebeat module and via Logstash end up in an ElasticSeach index - a very common scenario I would think.
And so, being able to build on top of the metadata key-values pairs for searching and filtering etc. makes it extremely useful to have these values not be a part of the actual log statement.

Any suggestions or hints as to I could accomplish this?

Question about the tags concept:

I have read the explanations about Metadata and Tags but I am however a bit confused about the use of the word Tag to describe a key/value pair.
I would have assumed a tag would be only a single value such as `SystemYTag` and could be included in the output like this:

```json
{
"logger_name":"mypackage.SpringBootApplication",
"metadata":{
"myTimestampMS":1643009127300
},
"tags": ["tagA", "tagB"]
}
```

I would think it could be possbile then, to use the term `labels` to describe those values but the `label` term is used as a name for MetadataKey names and as well in LoggingScope and ValueQueue to name a few.
Maybe `LogLabels` could be a candidate. :)

Environment info:

I am running the test using JDK 8, a Spring Boot service based on 2.2.9 and the Logback configuration files used in this Spring Boot version can be found here:
[Spring Boot Logback config files](https://github.com/spring-projects/spring-boot/tree/v2.2.9.RELEASE/spring-boot-project/spring-boot/src/main/resources/org/springframework/boot/logging/logback).

I am using the following logging relevant libs/versions:
* com.google.flogger:flogger:jar:0.7.4
* com.google.flogger:flogger-system-backend:jar:0.7.4
* com.google.flogger:flogger-slf4j-backend:jar:0.7.4
* ch.qos.logback:logback-classic:jar:1.2.3:compile
* net.logstash.logback:logstash-logback-encoder:jar:5.3:compile

I hope I am getting my use case and my point across clearly - otherwise please ask me to clarify further.

With regards to building the Flogger project, I also have a small question: How do I build it? :)
I do not have experience with Bazel but installed it and got it running.
However, I could no immediately see how I am supposed to build the entire project as I would normally do with a multi-module project in Maven.
Any hints would be much appreciated.
I would like to try out making some changes locally and so building the project and installing the jars locally will make that a bit easier. ;)

Also, if I were to post questions related to Flogger on StackOverflow, which tag(s) should I use?

Thank you in advance
Henrik

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.