Deduplicating processing by reusing processed output

### What problem does your feature request solve?

As we run large CSET runs, and especially those with case aggregation, etc., performance is starting to become a concern. We currently write out the data after a processing run and has been proposed to reuse this data when aggregating, as that would save significant compute/IO time.

### Describe the solution you'd like

It would be nice to reuse this processing, however we also want to minimise the interdependency of the cylc tasks to maintain parallelism.

### Describe alternatives you've considered

We could move to running all recipes inside the same process, then using dask or similar to deduplicate the tasks required. This would cost us multi-machine parallelism, but would allow us to make better use of many core machines, and would significantly reduce overall compute time.

Start by tracing how processing outputs are written and how Cylc aggregation tasks consume them. Compare reusing those outputs with the alternative of running recipes in one process using Dask or similar. Done should mean duplicated processing and I/O are reduced while multi-machine parallelism is preserved as far as practical.

Written by the indexing model from the issue text.