killme2008 / killme2008/xmemcached

Memcache timeout exceptions still seen

Open
#102 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
764
Forks
271
PR merge metrics
No merged PRs in 30d

Description

Referece: https://github.com/killme2008/xmemcached/issues/30

We picked up https://mvnrepository.com/artifact/com.googlecode.xmemcached/xmemcached/2.4.5 and we still see the exception. Here is the exception trace we continue to see in our labs where we upgraded the client jar xmemcached-2.4.5.jar.
site-2-pps06 site-2-pps06 2019-03-18 08:42:12,943 [pool-3-thread-1] WARN c.b.memcache.impl.MemcacheClient - Memcached failure
java.util.concurrent.TimeoutException: Timed out(150 milliseconds) waiting for operation while connected to 172.26.2.61:11211
at net.rubyeye.xmemcached.XMemcachedClient.latchWait(XMemcachedClient.java:2380) ~[xmemcached-1.3.8.jar:na]
at net.rubyeye.xmemcached.XMemcachedClient.sendStoreCommand(XMemcachedClient.java:2345) ~[xmemcached-1.3.8.jar:na]
at net.rubyeye.xmemcached.XMemcachedClient.add0(XMemcachedClient.java:1337) ~[xmemcached-1.3.8.jar:na]
at net.rubyeye.xmemcached.XMemcachedClient.add(XMemcachedClient.java:1331) ~[xmemcached-1.3.8.jar:na]
at net.rubyeye.xmemcached.XMemcachedClient.add(XMemcachedClient.java:1360) ~[xmemcached-1.3.8.jar:na]
at net.rubyeye.xmemcached.XMemcachedClient.add(XMemcachedClient.java:1349) ~[xmemcached-1.3.8.jar:na]
at com.broadhop.memcache.impl.MemcacheClient.add(MemcacheClient.java:281) ~[com.broadhop.common.service_14.0.1.r133471.jar:na]
at com.broadhop.memcache.impl.SemaphoreManager.getSemaphore(SemaphoreManager.java:73) [com.broadhop.common.service_14.0.1.r133471.jar:na]
at sun.reflect.GeneratedMethodAccessor_RN2F290B39.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0-zing_18.06.0.0]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0-zing_18.06.0.0]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309) [org.springframework.aop_3.0.6.RELEASE.jar:3.0.6.RELEASE]
at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:58) [org.springframework.osgi.core_2.0.0.M1.jar:2.0.0.M1]
at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:62) [org.springframework.osgi.core_2.0.0.M1.jar:2.0.0.M1]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [org.springframework.aop_3.0.6.RELEASE.jar:3.0.6.RELEASE]
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131) [org.springframework.aop_3.0.6.RELEASE.jar:3.0.6.RELEASE]
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119) [org.springframework.aop_3.0.6.RELEASE.jar:3.0.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [org.springframework.aop_3.0.6.RELEASE.jar:3.0.6.RELEASE]
at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:71) [org.springframework.osgi.core_2.0.0.M1.jar:2.0.0.M1]
at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:54) [org.springframework.osgi.core_2.0.0.M1.jar:2.0.0.M1]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [org.springframework.aop_3.0.6.RELEASE.jar:3.0.6.RELEASE]
at org.springframework.osgi.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:59) [org.springframework.osgi.core_2.0.0.M1.jar:2.0.0.M1]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [org.springframework.aop_3.0.6.RELEASE.jar:3.0.6.RELEASE]
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131) [org.springframework.aop_3.0.6.RELEASE.jar:3.0.6.RELEASE]

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the referenced xmemcached issue and the XMemcachedClient.latchWait and sendStoreCommand frames in the supplied trace. Compare the reported xmemcached-2.4.5 upgrade with the trace, which shows xmemcached-1.3.8, and investigate the 150-millisecond timeout during add. Done means the cause is identified and the reported timeout behavior is corrected or clearly reproduced and documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, memcached
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.