killme2008 / killme2008/xmemcached

XMemcached pooled client Recv Q size is very high

Open
#77 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
764
Forks
271
PR merge metrics
No merged PRs in 30d

Description

Hi,

We've been using XMemcachedClient connected to twemproxy which sits behind set of memcached servers. We've kept a OpTimeout of 30 ms.

Over the time, the Recv Q(from netstat) size of the tcp socket grows really high and prompting timeouts from XMemcached. Twemproxy as a result is also getting killed by going OOM gradually

Is XMemcached client slow in receiving data? Should `TCP_RECV_BUFF_SIZE` be increased?

Few more details:
1. XMemcached version: 2.4.0
2. Protocol: Text Protocol
3. Session Idle Timeout: 60 secs
4. timeoutExceptionThreshold: 250000
5. healSessionInterval: 1 sec
6. Pool Size: 20
7. Throughput: ~ 2.2K QPS

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the XMemcachedClient pooled-client behavior described for version 2.4.0, using the twemproxy, text-protocol, 30 ms timeout, and 2.2K QPS conditions as the reproduction context. Inspect the TCP receive queue, timeout behavior, and TCP_RECV_BUFF_SIZE handling; done means identifying the cause of queue growth and documenting or implementing a validated fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, memcached
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.