spring-projects / spring-projects/spring-data-redis
LettuceConnectionProvider getConnection hang forever
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.9k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
We use spring-data-redis and lettuce to access the redis cluster.
I don't know the cause, but there was a problem that the connection could not be initialization complete
As a result, sharedConnection could not be inited and fell into a waiting state forever
And after that, all requests became blocking.
###thread dump
"lettuce-epollEventLoop-5-6" tid=0x3f native=false suspended=false
java.lang.Thread.State: WAITING
at jdk.internal.misc.Unsafe.park(java.base@11.0.9.1/Native Method)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.9.1/LockSupport.java:194)
at java.util.concurrent.CompletableFuture$Signaller.block(java.base@11.0.9.1/CompletableFuture.java:1796)
at java.util.concurrent.ForkJoinPool.managedBlock(java.base@11.0.9.1/ForkJoinPool.java:3128)
at java.util.concurrent.CompletableFuture.waitingGet(java.base@11.0.9.1/CompletableFuture.java:1823)
at java.util.concurrent.CompletableFuture.get(java.base@11.0.9.1/CompletableFuture.java:1998)
at io.lettuce.core.cluster.RedisClusterClient.get(RedisClusterClient.java:937)
at io.lettuce.core.cluster.RedisClusterClient.getPartitions(RedisClusterClient.java:329)
at org.springframework.data.redis.connection.lettuce.ClusterConnectionProvider.getConnectionAsync(ClusterConnectionProvider.java:92)
at org.springframework.data.redis.connection.lettuce.ClusterConnectionProvider.getConnectionAsync(ClusterConnectionProvider.java:40)
at org.springframework.data.redis.connection.lettuce.LettuceConnectionProvider.getConnection(LettuceConnectionProvider.java:53)
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$ExceptionTranslatingConnectionProvider.getConnection(LettuceConnectionFactory.java:1527)
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.getNativeConnection(LettuceConnectionFactory.java:1315)
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.getConnection(LettuceConnectionFactory.java:1298)
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getSharedReactiveConnection(LettuceConnectionFactory.java:1049)
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getReactiveClusterConnection(LettuceConnectionFactory.java:481)
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getReactiveConnection(LettuceConnectionFactory.java:457)
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getReactiveConnection(LettuceConnectionFactory.java:101)
at org.springframework.data.redis.core.ReactiveRedisTemplate.lambda$doInConnection$0(ReactiveRedisTemplate.java:198)
at org.springframework.data.redis.core.ReactiveRedisTemplate$$Lambda$1725/0x0000000100cadc40.get(Unknown Source)
at reactor.core.publisher.MonoSupplier.call(MonoSupplier.java:85)
"lettuce-epollEventLoop-5-5" tid=0x3e native=false suspended=false
java.lang.Thread.State: BLOCKED
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.getConnection(LettuceConnectionFactory.java:1297)
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getSharedReactiveConnection(LettuceConnectionFactory.java:1049)
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getReactiveClusterConnection(LettuceConnectionFactory.java:481)
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getReactiveConnection(LettuceConnectionFactory.java:457)
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getReactiveConnection(LettuceConnectionFactory.java:101)
at org.springframework.data.redis.core.ReactiveRedisTemplate.lambda$doInConnection$0(ReactiveRedisTemplate.java:198)
at org.springframework.data.redis.core.ReactiveRedisTemplate$$Lambda$1725/0x0000000100cadc40.get(Unknown Source)
at reactor.core.publisher.MonoSupplier.call(MonoSupplier.java:85)
at reactor.core.publisher.FluxUsingWhen.subscribe(FluxUsingWhen.java:80)
at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:236)
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:203)
at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onComplete(ScopePassingSpanSubscriber.java:102)
at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FluxSwitchIfEmpty.java:84)
at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onComplete(ScopePassingSpanSubscriber.java:102)
at reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber.onComplete(MonoIgnoreElements.java:88)
at reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber.onComplete(MonoIgnoreElements.java:88)
at reactor.core.publisher.Operators.complete(Operators.java:136)
We are looking for the cause of the connection failing to connect.
My guess is that getConnection should have a timeout.
https://github.com/spring-projects/spring-data-redis/blob/d2cae7528b84647b3b9fad266ab9f8245b7aa3ed/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceConnectionProvider.java#L53
https://github.com/spring-projects/spring-data-redis/blob/d2cae7528b84647b3b9fad266ab9f8245b7aa3ed/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceFutureUtils.java#L68
This is because it should not "hang" when sharedConnection init fails for various reasons.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with LettuceConnectionProvider.java at getConnection and LettuceFutureUtils.java at the referenced line, then compare those paths with the supplied thread dump. Reproduce or trace the failed sharedConnection initialization and determine how connection failure should stop waiting rather than block requests forever; done means the failure no longer hangs indefinitely.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, redis
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100