Resource Groups for sources

This is a new feature request.

Current functionality

WebAPI leverages spring batch to execute asynchronous jobs, which is handled by a single ThreadPoolTaskExecutor. There is a default thread pool size set in pom.xml, and this controls the maximum number of WebAPI jobs that can be running at one time. This is important for large scale DBMS (such as PDW and Redshift) where there are concurrency limits, and you want to ensure that the WebAPI jobs do not overwhelm the server.

However, the problem is that if you have multiple PDW or Redshift clusters, the single ThreadPoolTaskExecutor controls all access to all sources, meaning that filling up the thread pool with only Redshift jobs means that no threads are available for PDW. It would be better if PDW and Redshift (or the sources associated to PDW or Redshift) had their own ThreadPoolTaskExecutor.

Proposed functionality

The proposed changes would introduce a ResourceGroup or SourceResourceGroup (to be more specific) that would allocate resources (thread pools, caches, etc) that can be shared across sources within a SourceResourceGroup. Initially, this would be to create the ThreadPoolTaskExector that would handle spring batch jobs on a per-source basis, but could be extended to support other options later.

Implementation details

New Entity: SourceResourceGroup

Update Entity: Source

WebAPI Startup:
When WebAPI starts, it reads in all the SourceResourceGroup entities, and constructs/allocates the resources required, including ThreadPoolTaskExecutors.

Spring Boot Tasklet Changes:
When executing a spring batch tasklet, if the tasklet is targeting a source, it should use the ThreadPoolTaskExecutor associated to the source. if there is none (null), then use the default.

Questions

Should there be one SourceResourceGroup entity, with a flat table for all resource groups, or should the SourceResourceGroup be a more 'nested' entity where you have @Elements for data structures for ThreadPoolTaskExecutors, and any other construct (ie if you wanted to set a list of things like a list of caches for a given source, you would have to hack something to make it work in a single-tabe form).

Other than ThreadPoolTaskExectors, what other options may we needed when thinking about source-specific resources?

Start with pom.xml to understand the current ThreadPoolTaskExecutor settings, then trace WebAPI startup and the Spring Boot tasklet execution path described in the issue. Define the SourceResourceGroup and nullable Source relationship, allocate per-group executors at startup, and verify that source-targeted tasklets use the group executor while others use the default.