Implement end-to-end workflow tests

### What problem does your feature request solve?

Currently the cylc workflow is largely untested. While we do regularly do manual runs, these often miss out interesting combinations of diagnostics that can potentially cause issues. In the past this has lead to critical issues being released, necessitating a quick patch release when noticed.

### Describe the solution you'd like

We should add end-to-end smoke tests of the CSET cylc workflow.

Eventually we should aim to cover all of the interesting functionality that the workflow offers, however initially I'm interested in two cases.

1. A minimal run with just a single basic recipe enabled. This is to check the initial functioning of the workflow, as well as catch cases like #1840 where the workflow fails with a setting disabled.
2. A full run with everything we can enabled, to test that all the functionality can work.

Given required data and model types, we might have to split the second case into multiple runs.

As these tests will require cylc to be setup, a reasonable amount of compute, and potentially a large amount of input data, I'm not expecting to run these tests in CI, especially not every commit. Instead, I foresee these tests as something that can be manually (but easily) run locally via a `make` subcommand.

Consideration will have to be made towards making them possibly portable, instead of just Met Office specific. This will likely require at minimum being able to specify the value of the `SITE` variable to the test harness.

### Describe alternatives you've considered

~~Pain and suffering.~~ Increasing unit/integration testing for each of the individual components. This is probably good to do as well, but end-to-end smoke tests are likely a better investment of time.