Healthcheck Protocol Bug / Edge Case when Unhealthy or Unresponsive

We're facing an issue where the Invoker becomes Unhealthy (not processing activations), but it is still sending pings to the controller. This works as intended that the controller marks it as unhealthy or unresponsive. However, it is still sending healthcheck messages on the invoker topic which are not being read when the invoker is in this state. Coincidentally the other eager marking as healthy bug that was identified this week probably isn't helping too. The problem then starts on the restart of the invoker which brings the invoker back to a healthy state:

1. There is a huge backlog of healthcheck messages on the invoker topic (say it took days before the invoker restart)
2. After the invoker restarts, it starts processing these healthchecks finally.
3. Invoker tells the controller it's healthy from the first healthcheck messages processing.
4. Controller starts sending activations onto an invoker topic with a huge healthcheck backlog
5. These activations ultimately timeout, which results in the host unresponsive.
6. Then more healthchecks are completed from the backlog and back to healthy.
7. Sends more activations. Controller flaps the invoker between unresponsive and healthy until the backlog completes.

The easiest solution I thought for this would be to just when the invoker consumes an activation from kafka and it is a healthcheck to drop the message if it is older than a few minutes. This should be fine since once the controller views it as unhealthy or unresponsive, it sends a new healthcheck once a minute. The one potential issue with this is time drift. If the controller internal clock is behind the invoker by more than the threshold, than the invoker will drop every healthcheck and never become healthy. This might be edge casey enough though that we don't care.

As to why things are no longer being read off kafka, I'm not sure yet. I think there might be some edge case bugs with container management in the ContainerPool and ContainerProxy fsm logic. My theory is that there is a dead lock with requesting more activation messages from the feed when the invoker is under heavy load.

I have my eyes on this commit right now when the `processBufferOrFeed` function was introduced for reading messages from the feed since it's somewhat recent; looking for something, but haven't been able to identify anything yet. I'll update this issue as I find anything of concern in the code.

https://github.com/apache/openwhisk/commit/a7482ca22694eee9b578c1003cd2c847953273e8

Start with the processBufferOrFeed function introduced in commit a7482ca22694eee9b578c1003cd2c847953273e8, then inspect the ContainerPool and ContainerProxy FSM logic. Reproduce the restart scenario with a healthcheck backlog and determine why feed messages stop being read under load. Done means stale healthchecks cannot make the controller flap the invoker or send activations into a backlog.

Written by the indexing model from the issue text.