killme2008 / killme2008/xmemcached
TimeoutException causes leakage of connections in several servers
- Dominant language
- Java
- Stars
- 764
- Forks
- 271
- PR merge metrics
- No merged PRs in 30d
Description
After connection timeout from the initialization, failed address will be added to a waiting queue:
```
this.connector.addToWatingQueue(
new ReconnectRequest(inetSocketAddressWrapper, 0, getHealSessionInterval()));
log.error("Connect to " + SystemUtils.getRawAddress(inetSocketAddress) + ":"
+ inetSocketAddress.getPort() + " fail", throwable);
```
Stacktrace of the exception:
```
java.util.concurrent.TimeoutException: null
at com.google.code.yanf4j.core.impl.FutureImpl.get(FutureImpl.java:143) ~[xmemcached-2.4.7.jar:?]
at net.rubyeye.xmemcached.XMemcachedClient.connect(XMemcachedClient.java:565) [xmemcached-2.4.7.jar:?]
at net.rubyeye.xmemcached.XMemcachedClient.(XMemcachedClient.java:840) [xmemcached-2.4.7.jar:?]
at net.rubyeye.xmemcached.XMemcachedClientBuilder.build(XMemcachedClientBuilder.java:362) [xmemcached-2.4.7.jar:?]
```
such code in MemcachedConnector.java causes the infinite loop:
```
try {
log.info("Trying to connect to " + address.getAddress().getHostAddress() + ":"
+ address.getPort() + " for " + request.getTries() + " times");
if (!future.get(MemcachedClient.DEFAULT_CONNECT_TIMEOUT, TimeUnit.MILLISECONDS)) {
connected = false;
} else {
connected = true;
}
} catch (TimeoutException e) {
future.cancel(true);
} catch (ExecutionException e) {
future.cancel(true);
} finally {
if (!connected) {
this.rescheduleConnectRequest(request);
} else {
continue;
}
}
}
```
When future.get(MemcachedClient.DEFAULT_CONNECT_TIMEOUT, TimeUnit.MILLISECONDS) timed out 60 seconds , TimeoutException will be thrown ,and future.cancel(true) is called.
But , the underlying connections is actually established fron the netstat , and the cancellation didn't really cancel the connection. So the connection size keeps growing, there were 2000+ ESTABLISHED connections to a single destination, even if connection pool config is default(1).
Network delay is actually within 10ms.
Maybe somewhere blocked in Reactor?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in MemcachedConnector.java at the reconnect loop and trace the timeout path through FutureImpl.get and XMemcachedClient.connect. Reproduce a connection timeout while observing established connections, then verify that retries do not accumulate connections beyond the configured pool size and that normal reconnect behavior remains intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100