filter+output pipeline workers watchdog

Followup on #7763

After discussing and brainstorming with @jordansissel and @andrewvc about the stalled/blocked filters or outputs plugins issues raised in #7763 we agreed for the following:

- We realize there is a difficulty in differentiating between an unhealthy stalled plugin and a blocked plugin (typically an output plugin) for *legitimate* reasons to initiate back pressure propagation. For example, an ES output plugin which blocks until it can continue outputting to ES, versus, conceptually, the grok filter which can stall in the regex parsing (ignoring the added timeout).

- Plugins stalls or blocks could be initiated by external conditions, for example a synchronous blocking IO call or could be related to the content of an event in which case this is probably a poison event.

- A single plugin not showing any activity is not an indication of a stall, depending on the configuration and the conditional expressions, a plugin could receive rare/few or no event to process which is not a problem.

- As a whole, a pipeline worker not processing any event for an arbitrary period while events are being processed by other workers or the queue is non-empty is an indication of a potential unhealthy stall or block, somewhere in the filters or outputs.

- We already have a mechanism in place to report stalls/blocks but only at shutdown time, because this is usually where such a stall has a visible effect: it prevents the shutdown sequence to complete. The rationale was to at least provide feedback to the user about the condition.

### Proposal

As a first step, we agreed that we could generalize the concept of the shutdown reporter and make a pipeline workers watchdog with the only task, for now, of notifying/logging any worker stall condition. The period to consider a worker in a stall condition could be configurable.

The benefit is that this will add visibility about a condition that is currently completely opaque. There is no easy way to see that a worker is stuck in any way now. It will provide very good insights about the health of the workers to help debug problems, not just at shutdown time but also during the normal runtime.

The good news is this idea does not require any plugin refactor or modifications. Conceptually it is relatively simple and can be done in the current pipeline in or around the worker loop.

Once this is in place and we gather more information we will see what the possible next steps are. For example if we can/should add a DLQ+worker "kill" mechanism or other functionality to help mitigate stall conditions without having to shutdown.