RFC: Implement a runner for fast background jobs

How to use GitHub

• Please use the 👍 reaction to show that you are interested into the same feature.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

Is your feature request related to a problem? Please describe.
Currently, some apps require the setup of a fast background job worker using occ background-job:worker. Some apps require more than one worker of the same job, even. Setting these up e.g. with systemd is cumbersome and repetitive. Additionally for nextcloud AIO, it would be beneficial to have a generic way to run these without hard coding jobs for all possible apps in the AIO code.

Describe the solution you'd like
We could have a new php script next to cron.php and occ or just another occ command that

• figures out which jobs to run
• runs the necessary background workers as sub processes
• keeps track of them and restarts them when they fail or exit
• Passes the stdout of the workers through to its own stdout

Background services could be set via a a php file, or a json file. A service could be either a background-job:worker "worker" or a "dedicated-cron" cron.php runner, with a set list of jobs. This would cater to both QueuedJobs and TimedJobs (the latter cannot be run by the bg-job worker).

{
  "services": {
    "TaskProcessing 1": {
      "type": "worker",
      "jobs": ["OC\\TaskProcessing\\SynchronousBackgroundJob"],
    },
   "TaskProcessing 2": {
      "type": "worker",
      "jobs": ["OC\\TaskProcessing\\SynchronousBackgroundJob"],
    },
    "WebhookListeners": {
      "type": "worker",
      "jobs": ["OCA\\WebhookListeners\\BackgroundJobs\\WebhookCall"],
    },
    "ContextChat": {
      "type": "dedicated-cron"
      "jobs": ["OCA\\ContextChat\\BackgroundJobs\\CrawlJob", "OCA\\ContextChat\\BackgroundJobs\\IndexerJob"],
    }
  }
}

Services would be started only if the respective job classes are loadable in the main php process.

Describe alternatives you've considered
We could also leave out the dedicated cron feature, which would make the whole thing less powerful but still useful.

Additional context

Start by reading the existing occ background-job:worker entry point and cron.php, which the issue identifies as the relevant runners. Define how a new runner would load configured services, launch and restart worker subprocesses, and forward stdout; done means the proposed worker and dedicated-cron modes handle loadable jobs reliably.