spring-projects / spring-projects/spring-data-redis

LettuceConnectionProvider getConnection hang forever

Open
#2,289 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

theme: 4.0 type: enhancement
Dominant language
Java
Stars
1.9k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

We use spring-data-redis and lettuce to access the redis cluster.

I don't know the cause, but there was a problem that the connection could not be initialization complete
As a result, sharedConnection could not be inited and fell into a waiting state forever
And after that, all requests became blocking.

###thread dump

"lettuce-epollEventLoop-5-6" tid=0x3f native=false suspended=false
   java.lang.Thread.State: WAITING
	at jdk.internal.misc.Unsafe.park(java.base@11.0.9.1/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.9.1/LockSupport.java:194)
	at java.util.concurrent.CompletableFuture$Signaller.block(java.base@11.0.9.1/CompletableFuture.java:1796)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@11.0.9.1/ForkJoinPool.java:3128)
	at java.util.concurrent.CompletableFuture.waitingGet(java.base@11.0.9.1/CompletableFuture.java:1823)
	at java.util.concurrent.CompletableFuture.get(java.base@11.0.9.1/CompletableFuture.java:1998)
	at io.lettuce.core.cluster.RedisClusterClient.get(RedisClusterClient.java:937)
	at io.lettuce.core.cluster.RedisClusterClient.getPartitions(RedisClusterClient.java:329)
	at org.springframework.data.redis.connection.lettuce.ClusterConnectionProvider.getConnectionAsync(ClusterConnectionProvider.java:92)
	at org.springframework.data.redis.connection.lettuce.ClusterConnectionProvider.getConnectionAsync(ClusterConnectionProvider.java:40)
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionProvider.getConnection(LettuceConnectionProvider.java:53)
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$ExceptionTranslatingConnectionProvider.getConnection(LettuceConnectionFactory.java:1527)
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.getNativeConnection(LettuceConnectionFactory.java:1315)
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.getConnection(LettuceConnectionFactory.java:1298)
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getSharedReactiveConnection(LettuceConnectionFactory.java:1049)
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getReactiveClusterConnection(LettuceConnectionFactory.java:481)
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getReactiveConnection(LettuceConnectionFactory.java:457)
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getReactiveConnection(LettuceConnectionFactory.java:101)
	at org.springframework.data.redis.core.ReactiveRedisTemplate.lambda$doInConnection$0(ReactiveRedisTemplate.java:198)
	at org.springframework.data.redis.core.ReactiveRedisTemplate$$Lambda$1725/0x0000000100cadc40.get(Unknown Source)
	at reactor.core.publisher.MonoSupplier.call(MonoSupplier.java:85)
"lettuce-epollEventLoop-5-5" tid=0x3e native=false suspended=false
   java.lang.Thread.State: BLOCKED
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.getConnection(LettuceConnectionFactory.java:1297)
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getSharedReactiveConnection(LettuceConnectionFactory.java:1049)
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getReactiveClusterConnection(LettuceConnectionFactory.java:481)
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getReactiveConnection(LettuceConnectionFactory.java:457)
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getReactiveConnection(LettuceConnectionFactory.java:101)
	at org.springframework.data.redis.core.ReactiveRedisTemplate.lambda$doInConnection$0(ReactiveRedisTemplate.java:198)
	at org.springframework.data.redis.core.ReactiveRedisTemplate$$Lambda$1725/0x0000000100cadc40.get(Unknown Source)
	at reactor.core.publisher.MonoSupplier.call(MonoSupplier.java:85)
	at reactor.core.publisher.FluxUsingWhen.subscribe(FluxUsingWhen.java:80)
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:236)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:203)
	at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onComplete(ScopePassingSpanSubscriber.java:102)
	at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FluxSwitchIfEmpty.java:84)
	at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onComplete(ScopePassingSpanSubscriber.java:102)
	at reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber.onComplete(MonoIgnoreElements.java:88)
	at reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber.onComplete(MonoIgnoreElements.java:88)
	at reactor.core.publisher.Operators.complete(Operators.java:136)

We are looking for the cause of the connection failing to connect.
My guess is that getConnection should have a timeout.
https://github.com/spring-projects/spring-data-redis/blob/d2cae7528b84647b3b9fad266ab9f8245b7aa3ed/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceConnectionProvider.java#L53
https://github.com/spring-projects/spring-data-redis/blob/d2cae7528b84647b3b9fad266ab9f8245b7aa3ed/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceFutureUtils.java#L68
This is because it should not "hang" when sharedConnection init fails for various reasons.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with LettuceConnectionProvider.java at getConnection and LettuceFutureUtils.java at the referenced line, then compare those paths with the supplied thread dump. Reproduce or trace the failed sharedConnection initialization and determine how connection failure should stop waiting rather than block requests forever; done means the failure no longer hangs indefinitely.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, redis
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.