spring-cloud / spring-cloud/spring-cloud-stream
Lots of open kafka-producer-network-thread after RefreshRemoteApplicationEvent
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.1k
- Forks
- 646
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 8
Description
Description
I've built a Spring Cloud Config Server integrated with Spring Cloud Bus via Kafka for dynamic property refreshing.
I've another application that consumes these properties and refreshes them dynamically.
Endpoint /monitor in Spring Cloud Config Server is used to update the service properties.
The service receives RefreshRemoteApplicationEvent in springCloudBus kafka topic.
After several properties updates, we started getting the error
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
I noticed that a lot of kafka-producer-network-thread threads remain open and their number grows after each update of the properties.
I see this on both an Spring Cloud Config Server and a configurable service.
I suspect this is due to improper closing of Kafka connections after getting RefreshRemoteApplicationEvent
https://github.com/spring-cloud/spring-cloud-stream/commit/24c1abea50cb7016a138d0b33382c8f20f6ad954#diff-5e17c4149c35b951bd7e88675c6fab6589366819948092156fd0b70573f25632R370
During connection cache cleanup, the connection is not closed beforehand, and kafka-producer-network-thread threads continue to exist in the OS.
Version of the framework
org.springframework.boot:spring-boot-starter-parent:3.3.8
spring.cloud.dependencies 2023.0.5 -> org.springframework.cloud:spring-cloud-stream:jar:4.1.5
To Reproduce
Send several /monitor request to Spring Cloud Config Server or /busrefresh request to configurable service.
To view the threads, use the command
jcmd <PID> Thread.print | grep -E '"main"|"RMI"|"GC"|prio=' --color=auto
You can observe an increasing number of kafka-producer-network-thread threads.
Possible solution
Close kafka connections before cache cleanup.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing repeated /monitor or /busrefresh requests and inspect RefreshRemoteApplicationEvent handling alongside the connection cache cleanup referenced in commit 24c1abea50cb7016a138d0b33382c8f20f6ad954. Done means Kafka connections are closed before cache cleanup and repeated refreshes no longer leave growing kafka-producer-network-thread counts; verify with the provided jcmd Thread.print command.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka, spring
- Domain
- backend, distributed-systems, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100