spring-cloud / spring-cloud/spring-cloud-stream

Lots of open kafka-producer-network-thread after RefreshRemoteApplicationEvent

Open
#3,116 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
1.1k
Forks
646
Avg merge
2d 3h
Merged PRs (30d)
8

Description

Description
I've built a Spring Cloud Config Server integrated with Spring Cloud Bus via Kafka for dynamic property refreshing.
I've another application that consumes these properties and refreshes them dynamically.

Endpoint /monitor in Spring Cloud Config Server is used to update the service properties.
The service receives RefreshRemoteApplicationEvent in springCloudBus kafka topic.

After several properties updates, we started getting the error
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached

I noticed that a lot of kafka-producer-network-thread threads remain open and their number grows after each update of the properties.
I see this on both an Spring Cloud Config Server and a configurable service.

I suspect this is due to improper closing of Kafka connections after getting RefreshRemoteApplicationEvent
https://github.com/spring-cloud/spring-cloud-stream/commit/24c1abea50cb7016a138d0b33382c8f20f6ad954#diff-5e17c4149c35b951bd7e88675c6fab6589366819948092156fd0b70573f25632R370
During connection cache cleanup, the connection is not closed beforehand, and kafka-producer-network-thread threads continue to exist in the OS.

Version of the framework
org.springframework.boot:spring-boot-starter-parent:3.3.8
spring.cloud.dependencies 2023.0.5 -> org.springframework.cloud:spring-cloud-stream:jar:4.1.5

To Reproduce
Send several /monitor request to Spring Cloud Config Server or /busrefresh request to configurable service.

To view the threads, use the command
jcmd <PID> Thread.print | grep -E '"main"|"RMI"|"GC"|prio=' --color=auto
You can observe an increasing number of kafka-producer-network-thread threads.

Possible solution
Close kafka connections before cache cleanup.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing repeated /monitor or /busrefresh requests and inspect RefreshRemoteApplicationEvent handling alongside the connection cache cleanup referenced in commit 24c1abea50cb7016a138d0b33382c8f20f6ad954. Done means Kafka connections are closed before cache cleanup and repeated refreshes no longer leave growing kafka-producer-network-thread counts; verify with the provided jcmd Thread.print command.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka, spring
Domain
backend, distributed-systems, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.