Store scheduler metrics

Nobody has claimed this yet.

It would be useful to store metrics from the scheduler such as how long each job took to run. Number of concurrent jobs over time throughout a run. Data in a format that makes it easy to plot scheduling decisions and runner utilisation over time.

It may be better to improve the logging information, and then calculate the metrics by parsing the logs? Perhaps add a log entry per job completion with start, end, and duration?

Alternatively, or potentially as well as, save that information and more to the scratch directory as individual job run meta data. This could also include results state, perhaps error logs on a job failure. This would allow us to recreate the summary JSON files from individual job state JSON files.

Start by tracing the scheduler and the existing logging and scratch-directory outputs described in the issue. Compare whether job completion data should be logged, stored as per-job metadata, or both, and define a format that supports plotting scheduler decisions and runner utilisation and can recreate the summary JSON files.

Written by the indexing model from the issue text.