apache / apache/logging-log4j2

Support for Custom Value Serializer in Log4jEventBuilder#addKeyValue() and Asynchronous Serialization

Open
#2,996 6 comments 0 reactions 0 assignees View on GitHub
waiting-for-maintainer
Dominant language
Java
Stars
3.6k
Forks
1.7k
Avg merge
21h 30m
Merged PRs (30d)
27

Description

> In the current `Log4jEventBuilder#addKeyValue(String key, Object value)` implementation, complex objects are serialized using `String.valueOf()`, which lacks flexibility. My suggestion is to allow users to provide a custom serializer that can handle more complex serialization needs

We can provide a `StringBuilderFormattable` specialization there. That is, we can change `addKeyValue()` implementation such that:

```java
String encodedValue = value instance StringBuilderFormattable
? encodeUsingStringBuilderFormattable((StringBuilderFormattable) value)
: toString(value);
keyValuePairs.put(key, encodedValue);
```

But,

* This doesn't solve the encoding problem. Assume `JsonTemplateLayout` is employed, it will encode the `Map` passed and this will result in `"kvPairs": {"yourKey": "{\"some-value\": \"you-already-JSONified\"}"}`. See?
* This encourages a bad-practice: assumption on the logging layout employed. (What if the effective layout is XML?)
* There is a better alternative, which I will elaborate on below.

> ... handling complex objects, especially when structured logging is required.
> ... complex serialization needs (e.g., converting objects to `JSON`)

There is so much to unpack here.

For one, logging as follows

```java
LOGGER.info("userContext: {}", toJSON(userContext));
```

is **a bad-practice**, because this implies an _assumption on the logging layout employed_, which contradicts with the separation-of-concerns practiced with the logging API vs. logging implementation separation. What if the used layout is XML? etc. Hence, we should avoid encoding payloads with assumptions on the employed layout. Instead, we should allow registering arbitrarily-typed parameters, if necessary, with sufficient encoding hints (e.g., extending from [`MultiFormatStringBuilderFormattable`](https://logging.apache.org/log4j/2.x/manual/messages.html#MultiFormatStringBuilderFormattable)) and let them be handled by the employed layout.

Currently, `addKeyValue()` admissions get registered to a `Map` – we should amend this with a `Map`. Though the bigger problem is how KV-pairs are passed to the `LogBuilder`:

```java
try (final Instance c = CloseableThreadContext.putAll(keyValuePairs)) {
logBuilder.log(message, arguments.toArray());
}
```

That is, `log4j-slf4j2-impl` hacks [auto-clearing thread context (i.e., `CloseableThreadContext`)](https://logging.apache.org/log4j/2.x/manual/thread-context.html#CloseableThreadContext) to pass the KV-pairs all the way down to the layout. This is due to the fact that `LogBuilder` of Log4j doesn't have a counterpart for arbitrary KV-pairs and the feature that comes closest to it we can leverage (and we did) is thread context. There are a couple of angles we can approach this problem from:

1. Implement accepting non-`String`-typed values in `CloseableThreadContext` – [LOG4J2-1648](https://issues.apache.org/jira/browse/LOG4J2-1648) proposes this. Registering instances of custom types in a thread context can cause memory leakage in Java EE container environments, though this should not be a problem for `CloseableThreadContext`.
2. Extend `LogBuilder` to accept custom KV-pairs

> I propose that the serialization of the value should happen asynchronously, offloading the potentially expensive serialization task to a separate thread. This will prevent blocking the thread that calls logging methods (such as `log.info()`), thus improving performance in high-throughput logging environments where structured logging is used extensively.

_"Improve performance in high-throughput logging environments [by offloading the log event encoding to a background thread]"_ this is a claim that cannot hold without given sufficient context. Given the context, most of the time it becomes a special case rather than a general one we can practice for every user. Nevertheless, if we can pass arbitrary objects in KV-pairs (and effectively allow the layout to take charge of the complete encoding effort), you can combine this with [asynchronous logging](https://logging.apache.org/log4j/2.x/manual/async.html#log4j2.formatMsgAsync) to effectively encode events in a thread different from the one invoking the logging API.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.