spring-cloud / spring-cloud/spring-cloud-gateway
Automatic Context Propagation fails to work with MDC logging in reactive API Gateway
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 4.9k
- Forks
- 3.5k
- Avg merge
- 20h 57m
- Merged PRs (30d)
- 8
Description
We have developed our API Gateway using Spring Cloud Gateway framework based on reactive paradigm using below libraries:
Spring Boot: 3.2.0
Spring Cloud: 2023.0.0
Spring: 6.1.1
Spring Cloud Gateway: 4.1.0
We are using Logback SLF4J library for logging purpose and MDC for injecting contextual information like request and session tracking id.
We observed that MDC contextual information is NOT appropriately logged in log messages in reactive app execution since ThreadLocal is used by MDC whereas SCG is thread-agnostic. Request and session tracking id were missing or obsolete ones printed in many messages, mainly when thread switch happens.
We tried to enable automatic context propagation provided by Micrometer context-propagation(1.1.2 version) library to propagate contextual info to ThreadLocals.
public static void main(String args[]) {
SpringApplication.run(GatewayApplication.class, args);
Hooks.enableAutomaticContextPropagation();
final String key = "TRACK_ID";
ContextRegistry.getInstance().registerThreadLocalAccessor(
key,
() -> MDC.get(key),
value -> MDC.put(key, value),
() -> MDC.remove(key)
);
}
And update the reactor context in the global filter in which the request tracking id is generated as below:
public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
final String reqTrackId = UUID.randomUUID().toString();
MDC.put("TRACK_ID", reqTrackId);
log.info("Request received with route: {}", exchange.getRequest().getPath());
return chain.filter(exchange).contextWrite(Context.of("TRACK_ID", reqTrackId));
}
After making above changes, the MDC contextual information seems to be appropriately written in log messages.
However, the same issue happens when an exception is thrown from one of the global filter and its handled by custom error handler which is implementing ErrorWebExceptionHandler interface.
We tried to develop minimal version project that can reproduce the issue (which is attached in this issue for reference) and identified below 2 scenarios:
- when below statement is added then request tracking id is:
a. written in logs written in global filters
b. missing in logs written in custom exception handlerreturn chain.filter(exchange).contextWrite(Context.of("TRACK_ID", reqTrackId)); - When contextWrite statement is removed, then request tracking id is
a. randomly missing in logs written in global filters
b. written in logs written in custom exception handler.
One possible way we found to fix the above issue was to update(do not remove key/value from MDC) the MDC key registration as below however it does not seems appropriate:
ContextRegistry.getInstance().registerThreadLocalAccessor(
key,
() -> MDC.get(key),
value -> MDC.put(key, value),
() -> MDC.remove("")
);
We also tried to make similar changes in spring webflux project and it does not seems to have any issues in custom exception handler messages since there was no scope for contextWrite operator.
Could you please check the attached sample project that reproduces the reported issue and provide any possible help to resolve the issue? Thanks in advance.
PS: This issue seems similar to below reported issue however this issue provides more details:
https://github.com/spring-cloud/spring-cloud-gateway/issues/3564
Sample Project:
sample-api-gateway.zip
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the attached sample project and inspect its main method, global filter, and custom ErrorWebExceptionHandler. Run the reproduction with and without the contextWrite statement, then compare the tracking ID in filter and exception-handler logs. Done means the reported MDC propagation behavior is understood and a consistent expected result is established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spring, spring-boot
- Domain
- api, backend, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100