apache / apache/doris

[Enhancement] Add MDC log traces

Open
#62,394 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Description

Image

How to use Flink MDC(jobid)
https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/advanced/logging/

```
Flink adds the following fields to [MDC](https://www.slf4j.org/api/org/slf4j/MDC.html) of most of the relevant log messages (experimental feature):

Job ID
key: flink-job-id
format: string
length 32
This is most useful in environments with structured logging and allows you to quickly filter the relevant logs.

The MDC is propagated by slf4j to the logging backend which usually adds it to the log records automatically (e.g. in [log4j2 json layout](https://logging.apache.org/log4j/2.x/manual/json-template-layout.html#event-template-resolver-mdc).

Log4j 2 JsonTemplateLayout [#](https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/advanced/logging/#log4j-2-jsontemplatelayout)
JsonTemplateLayout is a customizable, efficient, and garbage-free JSON generating layout. It encodes LogEvents according to the structure described by the JSON template provided.

The required jar log4j-layout-template-json is bundled in the flink-dist for convenience.

For example templates, see the [Event Templates](https://logging.apache.org/log4j/2.x/manual/json-template-layout.html#event-templates).

Log4j 2 PatternLayout [#](https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/advanced/logging/#log4j-2-patternlayout)
Alternatively, it can be configured explicitly - [log4j pattern layout](https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PatternLayout.html) might look like this:

[%-32X{flink-job-id}] %c{0} %m%n.
```

### Solution

```
MdcUtils.wrapRunnable( obid, () -> {} );
```

```java
scheduledExecutorService.scheduleWithFixedDelay(
MdcUtils.wrapRunnable(MdcUtils.asContextData(jobId), this::intervalFlush),
flushIntervalMs,
flushIntervalMs,
TimeUnit.MILLISECONDS);

```

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.