[RFC] Admin-tooling / Background job improvements

The problem

The current background job system isn't sufficient for many uses, for a developer it isn't possible to use it for anything that needs to happen as soon as possible (e.g. sending call notifications) or something that needs to happen at a specfic time (e.g. 6:00) which leads to various apps implementing their own workarounds.

For sysadmins the system is hard to get insight into, QueuedJobs leave no trace the moment they are done, it's hard to get insight into which jobs are using up resources and interpreting job state is difficult.

Requirements

Below is a list of desired improvements to the background job system, both from the developer perspective that wants to run their code in the background and from a sysadmin/support perspective that wants to get insight into the system

Developer requirements

• Run a job "asap"
• Run a job at a specific time/date
• Mark a job as retryable, with optional delay

Sysadmin requirements

• Better ui/command for listing jobs
  • show difference between desired run time/date and actual run time/date
  • See QueuedJobs that have finished, possibly leave them in the db for a while, marked as complete
    • Or move to a new table, which would also allow to have a list of old runs from repeating background jobs
  • Mark failed jobs
  • Have mapping between php processes and jobs
  • See which jobs are currently running
• Metrics/monitoring
  • total time (memory, io?) grouped by job class
  • delay between desired run time/date and actual, both for specific time and "asap" jobs

Sollutions?

Possible implementation ideas

• new table that logs job runs instead of jamming more data into oc_jobs
• https://github.com/nextcloud/server/issues/52629

Related: https://github.com/nextcloud/server/issues/52629 https://github.com/nextcloud/server/pull/52055