关于Comsumer重连问题

在生产环境运行过程中，偶尔因为网络问题，导致Consumer一直报下面的错误
2014-08-28 13:21:53 ERROR SimpleFetchManager - 获取消息失败,topic=Disc,partition=0
com.taobao.metamorphosis.exception.MetaOpeartionTimeoutException: Send message timeout in 10000 mills
at com.taobao.metamorphosis.client.consumer.SimpleMessageConsumer.fetch(SimpleMessageConsumer.java:372) ~[metamorphosis-client-1.4.6.2.jar:na]
at com.taobao.metamorphosis.client.consumer.SimpleFetchManager$FetchRequestRunner.processRequest(SimpleFetchManager.java:227) [metamorphosis-client-1.4.6.2.jar:na]
at com.taobao.metamorphosis.client.consumer.SimpleFetchManager$FetchRequestRunner.run(SimpleFetchManager.java:215) [metamorphosis-client-1.4.6.2.jar:na]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_45]
2014-08-28 13:21:53 ERROR SimpleFetchManager - 获取消息失败,topic=Disc,partition=0
com.taobao.metamorphosis.exception.MetaOpeartionTimeoutException: Send message timeout in 10000 mills
at com.taobao.metamorphosis.client.consumer.SimpleMessageConsumer.fetch(SimpleMessageConsumer.java:372) ~[metamorphosis-client-1.4.6.2.jar:na]
at com.taobao.metamorphosis.client.consumer.SimpleFetchManager$FetchRequestRunner.processRequest(SimpleFetchManager.java:227) [metamorphosis-client-1.4.6.2.jar:na]
at com.taobao.metamorphosis.client.consumer.SimpleFetchManager$FetchRequestRunner.run(SimpleFetchManager.java:215) [metamorphosis-client-1.4.6.2.jar:na]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_45]

中间也进行了rebalance，从日志看也成功了。
2014-08-28 13:21:43 INFO ConsumerZooKeeper - begin rebalancing consumer IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408526986226-6 try #0
2014-08-28 13:21:44 WARN ConsumerZooKeeper - Consumer IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676090-1 with {Config=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525677558-3], StatusPoll=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525679271-4], Disc=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676090-1], Event=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408526986226-6], Perform=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408526986104-5], CollectSched=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676825-2]} doesn't need to be rebalanced.
2014-08-28 13:21:44 WARN ConsumerZooKeeper - end rebalancing consumer IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676090-1 try #0
2014-08-28 13:21:44 WARN ConsumerZooKeeper - rebalance success.
2014-08-28 13:21:44 WARN ConsumerZooKeeper - Consumer IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676825-2 with {Config=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525677558-3], StatusPoll=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525679271-4], Disc=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676090-1], Event=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408526986226-6], Perform=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408526986104-5], CollectSched=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676825-2]} doesn't need to be rebalanced.
2014-08-28 13:21:44 WARN ConsumerZooKeeper - end rebalancing consumer IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676825-2 try #0
2014-08-28 13:21:44 WARN ConsumerZooKeeper - rebalance success.
2014-08-28 13:21:44 WARN ConsumerZooKeeper - Consumer IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408526986226-6 with {Config=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525677558-3], StatusPoll=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525679271-4], Disc=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676090-1], Event=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408526986226-6], Perform=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408526986104-5], CollectSched=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676825-2]} doesn't need to be rebalanced.
2014-08-28 13:21:44 WARN ConsumerZooKeeper - end rebalancing consumer IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408526986226-6 try #0
2014-08-28 13:21:44 WARN ConsumerZooKeeper - rebalance success.

但从实际日志打印来看，一直在包这种问题，网络异常到恢复也就半个小时时间，但是这个异常从上周五一直打印到现在，说明Consumer重连还是存在一些问题

Start by tracing SimpleMessageConsumer.fetch and SimpleFetchManager$FetchRequestRunner around the reported timeout, then compare that flow with ConsumerZooKeeper's rebalance logs. Reproduce a network interruption if possible and inspect what happens after rebalance succeeds. Done means the consumer recovers after connectivity returns without continuously logging fetch timeouts.

Written by the indexing model from the issue text.