apache / apache/logging-log4j2

In NoSql appender, be able to use both default fields and custom fields

Open
#1,751 1 comment 0 reactions 0 assignees View on GitHub
enhancement plugins
Dominant language
Java
Stars
3.6k
Forks
1.7k
Avg merge
21h 30m
Merged PRs (30d)
27

Description

Right now, the fields of the documents that the NoSql appender creates are fixed, unless you use a MapMessage and a MessageLayout. In this case you get the fields from the MapMessage but none of the default fields.

If you do this (Kotlin code in this example, but Java would have the same problem):

``` Kotlin
logger.info(MapMessage(mapOf("aa" to "11", "bb" to "22")))
```

You can either get this:
``` JSON
{
"_id": {
"$oid": "64f1fc8ec553ec3593274f30"
},
"level": "INFO",
"loggerName": "InjectedLogger.-1169081147",
"message": "aa=\"11\" bb=\"22\"",
"source": {
"className": "InjectedLogger",
"methodName": "execute",
"fileName": "Main.kt",
"lineNumber": 32
},
"marker": null,
"threadId": {
"$numberLong": "1"
},
"threadName": "main",
"threadPriority": 5,
"millis": {
"$numberLong": "1693580430801"
},
"date": {
"$date": "2023-09-01T15:00:30.801Z"
},
"thrown": null,
"contextMap": {},
"contextStack": [],
"additionalFields": {}
}
```

Or this:
``` JSON
{
"_id": {
"$oid": "64f1dbbddb2eb17fd4809d51"
},
"aa": "11",
"bb": "22",
"additionalFields": {}
}
```

I want this (order of the fields is unimportant):
``` JSON
{
"_id": {
"$oid": "64f1fc8ec553ec3593274f30"
},
"level": "INFO",
"loggerName": "InjectedLogger.-1169081147",
"message": "aa=\"11\" bb=\"22\"",
"source": {
"className": "InjectedLogger",
"methodName": "execute",
"fileName": "Main.kt",
"lineNumber": 32
},
"marker": null,
"threadId": {
"$numberLong": "1"
},
"threadName": "main",
"threadPriority": 5,
"millis": {
"$numberLong": "1693580430801"
},
"date": {
"$date": "2023-09-01T15:00:30.801Z"
},
"thrown": null,
"contextMap": {},
"contextStack": [],
"additionalFields": {},
"aa": "11",
"bb": "22",
}
```

I have been doing some analysis and I believe that this functionality would be very difficult to achieve with the provided extension points, as the code that creates the document layout can't be swapped from configuration.

Then it would be easy to create a custom Message or a Layout that can control the message field and set it to something more interesting than repeating the custom properties (not part of this request). StructuredDataMessage, for instance, already has a message property that can be used via a custom layout.

I propose creating a configuration option that would enable this behavior.

I have been looking at the code and, apart from creating the configuration option, only a minor change would be required [here](https://github.com/apache/logging-log4j2/blob/d61f12bbd6318c3d1e1f086a1e5fcef353d31fb4/log4j-core/src/main/java/org/apache/logging/log4j/core/appender/nosql/NoSqlDatabaseManager.java#L260):

``` Java
if (serializable instanceof MapMessage) {
// Add these three lines
if (newOption){
setFields(event, entity);
}
setFields((MapMessage) serializable, entity);
} else {
setFields(event, entity);
}
```

Another option (maybe more flexible) would be to move all the code that sets the document properties from the event (`writeInternal` and `setFields` methods from `NoSqlDatabaseManager`) to a new class that can be swapped via configuration. Something in the lines of a `NoSqlObjectLayout` that would be conceptually similar to Layouts but that outputs (or sets fields to) a NoSql document instead of a `String`.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in log4j-core/src/main/java/org/apache/logging/log4j/core/appender/nosql/NoSqlDatabaseManager.java, especially writeInternal and the setFields methods. Review how MapMessage handling and configuration options are implemented, then define the option's behavior and verify that default event fields and custom fields are both retained in the NoSQL document.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
observability-sre
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.