Improved Threading: Support timeouts with shared thread across processors

Is your feature request related to a problem? Please describe.

If a pipeline contains multiple Grok processors, each processor will require one single-thread from a thread pool. But, since they run in the same worker thread, they will never need to run in parallel.

Example pipeline:

source:
  file:
processors:
  - grok:
  - grok:
sink:
  - stdout:

Describe the solution you'd like

Create an interface which plugins can get from Data Prepper core which can run a method with a timeout. The actual implementation would be written to ensure that a single-thread thread pool is created per worker pipeline.

public interface TimeoutRunner {
  void run(Runnable runnable, long timeout, TimeUnit unit);
}

GrokProcessor would use it like:

if (grokPrepperConfig.getTimeoutMillis() == 0) {
    grokProcessingTime.record(() -> matchAndMerge(event));
} else {
    timeoutRunner.run(() -> grokProcessingTime.record(() -> matchAndMerge(event)), 
        grokPrepperConfig.getTimeoutMillis(), TimeUnit.MILLISECONDS);
}

The plugin framework can provide the concrete implementation to processors as a constructor parameter.

Describe alternatives you've considered (Optional)

1. Grok could be updated to stop using a thread pool and just run a single thread. Taking this route might make it harder to get a handle on the number of threads though.

2. This concept could be generalized to allow any Processor to declare that it needs N threads. Then Data Prepper could allocate a pool for the maximum needed. This approach might require more work than is currently needed. We would also need to consider what it means to request threads. Do we expect that they complete along with the processor? In my view, this alternative is more complicated than Data Prepper needs. The current proposal is compatible with this alternative, and may even serve as the base implementation. Data Prepper could support this more complex approach when requested.

Additional context

PR #708 fixed a related problem. Prior to this, the Grok processor had thread contention of sharing the same threads across workers. The solution in PR #708 created a single instance per worker thread. This proposal could allow Grok to share an instance across threads. This could allow for sharing some state such as compiled regex patterns.

Start by reviewing the GrokProcessor proposal in this issue and related PR #708, then trace how processors and worker pipelines are constructed. Define the plugin-facing TimeoutRunner contract and its per-worker pipeline behavior; done means Grok can use shared timeout execution without unnecessary thread allocation and the relevant behavior is covered by tests.