Consumer status remains in ERR value is equal to 3

Nobody has claimed this yet.


Recently our consumer group has been going constantly into status ERR due to which we are getting way much higher lag then the actual lag like 100X more which is not the actual lag for consumer groups.
{"error":false,"message":"consumer status returned","status":{"cluster":"prod","group":"","status":"ERR","complete":1}
this is the output of api of burrow.
I have also attached the snapshot that it has been oscillating between 1 and 3 but most of the time it remains at 3 which indicates that consumer group is in ERR state.
We tried at looking its resources which are as follows

This is way much higher then expected still we gave that much resources from our infra but issue is not resolved.
we have also tried for the single consumer group (by applying allowed list ) but the outcome was same.
its cpu usage usually go from 0.23 % to 1000%. in a second and that behaviour is constant.
Please suggest us where we can improve. @toddpalino. @timbertson
**NOTE:**
The behaviour is not same for all consumer groups some consumer group remains at 1 which indicate status OK.

Start with the consumer status API response and the reported status values of 1 and 3, then compare affected consumer groups with groups that remain at 1. Trace the status and lag calculation entry points to identify why the reported lag and CPU behavior diverge; done means the cause is reproducible and the status and lag accurately reflect the consumer group.

Written by the indexing model from the issue text.