Various Potential Bugs in Container Pool and Container Proxy

I've found what I believe are some potential cases and would like some confirmation that I'm in the right direction with these bugs before I start trying to remediate them. Would like to hear people's opinions on if any of these seem like they might actually be the case. These are just a few things I found after looking at these two files for a couple hours. If a lot of them are, we may want to audit the Container Pool and Container Proxy for edge cases as the logic in these FSM's and FSM management have gotten quite complex. This investigation was brought forth by the bug I opened earlier today about the healthcheck protocol and building healthcheck lag in the invoker topics.

1. On the failure of a cold start in the Container Proxy, we attempt to go to state `Removing`. I believe this results in the FSM from never being stopped as once in state `Removing` no new messages will be sent to it. The message that ultimately is sent to stop the FSM is sent from within the `destroyContainer` function and that will never be called in this case if concurrency is not used and there's only one activation in the proxy. This is the line that was added recently to move to that state on this failure:

https://github.com/apache/openwhisk/commit/4956362f8f9feb217d3c96feb644cdb2ffb7ee6c#diff-4b149cfaa5f4b88c5fe7d5fd798d4ff12ab655c7d4e4fccb47072b7763800a05R502

2. When the container pool is completely busy, it now relies on an event to hit that calls `processBufferOrFeed` to attempt to execute a new `Run` message. I'm afraid that this is error prone to dead locks even though not re-calling `Run` with the head of the buffer at the end of trying to schedule that message in the ContainerPool frees up CPU cycles. I haven't been able to pinpoint anything yet that would lead to a deadlock, but we are seeing behavior in our invokers where it appears that a deadlock is happening on loaded invokers rarely.

https://github.com/apache/openwhisk/commit/a7482ca22694eee9b578c1003cd2c847953273e8#diff-907fea4f2800af33d571845b19ddc9a24a44d481994305f344c23d33310d471aL205

3. We are seeing lots of error logs of `unhandled event Remove in state Removing`. This happens when the Container Pool sends a `Remove` message to the ContainerProxy when it's already in the `Removing` state in the fsm. My belief is that this is happening when the invoker is loaded and when some containers are failing it moves to removing early; and the container pool subsequently tries to remove the container to force a container recreation for a new action. This is probably fairly innocuous, but should be handled.

4. The `Removing` state used to stop the fsm if it received any failure message while in the state. The `destroyContainer` function forwards the `ContainerRemoved` message when all futured operations related to destroying the container completes which `Removing` receives and stops the fsm. On the unhappy case if something fails in `destroyContainer`, removing will no longer stop the fsm.

https://github.com/apache/openwhisk/commit/d4f9e8abf6739bd6ee48d64f567f93a90ef91daf#diff-4b149cfaa5f4b88c5fe7d5fd798d4ff12ab655c7d4e4fccb47072b7763800a05L541

5. This seems over eager to be calling `processBufferOrFeed`. Previously it would just try to dequeue the next item in the buffer, but by calling `processBufferOrFeed` you may be pulling from the message feed too soon since the buffer could now be empty:

https://github.com/apache/openwhisk/commit/a7482ca22694eee9b578c1003cd2c847953273e8#diff-907fea4f2800af33d571845b19ddc9a24a44d481994305f344c23d33310d471aR183

Start by reviewing the Container Pool and Container Proxy FSM code at the linked commit locations and tracing the five reported transitions. Confirm which cases reproduce under the stated concurrency and load conditions; done means a decided scope with focused fixes and regression coverage.

Written by the indexing model from the issue text.