Propagate CDDS sub-workflow failures and prevent stalled CDDS runs blocking subsequent CMEW workflows

Nobody has claimed this yet.

**Description**

CMEW launches a CDDS conversion as a sub-workflow (`cdds_CMEW`) during the `standardise_model_data `task and waits for it to complete. If the CDDS workflow stalls or fails, the parent CMEW workflow does not fail fast and continues running indefinitely.
Additionally, while a stalled or failed `cdds_CMEW `workflow remains present, new CMEW workflows cannot be started successfully without manual intervention (e.g. `cylc stop --now --now cdds_CMEW`). This creates operational coupling between CMEW runs and requires human cleanup to recover.
This behaviour was observed repeatedly during failures in the `standardise_model_data `task when CDDS conversion failed.
________________________________________
**Observed behaviour**
• CDDS (cdds_CMEW) fails or stalls.
• The parent CMEW task continues running and waits indefinitely.
• Subsequent CMEW workflows are blocked because an existing cdds_CMEW workflow is still present.
• Manual cleanup (cylc stop --now --now cdds_CMEW) is required before a new CMEW run can proceed.
___________________________________________
**Expected behaviour**
1. If the CDDS sub-workflow fails or stalls, the parent CMEW workflow should:

- Detect the failure promptly
- Fail the `standardise_model_data `task
- Stop or clean up the child CDDS workflow automatically

2. A failed or stalled CDDS run from one CMEW workflow should not block subsequent CMEW workflows.
________________________________________
**Impact**
- CMEW workflows can remain running indefinitely with no progress.
- New CMEW runs are operationally blocked by previous failed runs.
- Manual intervention is required to recover, reducing reliability and automation.
- Makes parallelisation and scaling of CMEW unsafe.
________________________________________
**Proposed implementation plan**
1. ~Fail-fast propagation from CDDS to CMEW~ Will be resolved via #158
• After launching the CDDS workflow, explicitly monitor its state.
• If the CDDS workflow enters a failed or stopped state, the CMEW task should exit non-zero and fail.
• Avoid masking CDDS failures by wrapper scripts.
________________________________________
2. ~Introduce a timeout watchdog for CDDS~ Captured in #158
• Add a configurable timeout (e.g. CDDS_TIMEOUT_HOURS).
• If CDDS does not complete within the timeout:
o Mark the CMEW task as failed
o Trigger cleanup logic
• Prevent infinite waiting states.
________________________________________
3. ~Ensure CDDS workflows are uniquely named per CMEW run~ Completed via #164
• Avoid using a fixed workflow name (cdds_CMEW) for all runs.
• Include a unique identifier (e.g. CMEW run ID, suite ID, or timestamp) in the CDDS workflow name.
• This prevents cross-run interference and allows safe parallel execution.
________________________________________
4. ~Automatic cleanup on failure~ Will be resolved via #519
• On CDDS failure or timeout:
o Automatically stop the child CDDS workflow
o Optionally clean its run directory if safe to do so
• Remove the need for manual cylc stop --now commands.
________________________________________
5. ~Improve logging and status visibility~ Captured in #158
• Log the CDDS workflow name, run directory, and elapsed wait time.
• Periodically report that CMEW is waiting on CDDS and its current state.
• Make it clear in logs and Cylc UI why CMEW is blocked.
________________________________________
**Acceptance criteria**
• CMEW fails cleanly if CDDS fails.
• A failed CDDS run does not block subsequent CMEW workflows: no manual cleanup is required to start a new CMEW run after a CDDS failure.
• Behaviour is documented and reproducible.
________________________________________
**Notes**
This work is a prerequisite for safely parallelising CDDS standardisation (see #315) and improves overall CMEW robustness and operational reliability.

Start at the standardise_model_data task and trace how it launches and waits for cdds_CMEW; review the failure, timeout, naming, and cleanup work referenced in #158, #164, and #519. Reproduce the stalled-child scenario if possible, then verify that a CDDS failure or timeout fails CMEW, cleans up the child, and allows a subsequent CMEW run without manual cylc stop intervention.

Written by the indexing model from the issue text.